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Abstract 


The purpose for this study was to find an assessment that is currently being used and is a 
common assessment familiar to others within the Brick-and-mortar classroom and 
transform that assessment for use within the online learning environment. By using 
assessments, teachers can measure whether or not students are learning what it is they are 
teaching. (Methodology) The purpose of this qualitative, comparative case study was to 
study the relationships between creating portfolios and how this process impacted 
students within an online graphic design (digital Imaging) course. (Results) With this 
study, the intent was to lay the foundation for using portfolios in virtually any graphic 
design course and to be able to see the progress of students through their portfolios each 
week. Another intent of this study was to measure how a student’s perceived self-efficacy 
changed during the process of creating a portfolio. (Conclusions) The uses of portfolios 
are common within the work force when trying to obtain a position with the “fine arts, 
architecture, fashion design, graphic artists, orthodontists and plastic surgeons” (Angelo 
and Cross, 1993, p. 1 14). Drummond (2004) provided one of the first portfolio 
assessments within an online science course, which resulted in starting and completing 
this study for an undergraduate, online, graphic design course and opens the door for 
future use within other online courses within Higher Education. (Contains 8 tables) 
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CHAPTER 1. INTRODUCTION 


Introduction to the Problem 

The nature of higher education has been undergoing a profound change. The 
paradigm shift away from the process to outcome orientation alters the direction of all 
higher education processes in Europe and America (Zawacki-Richter, Hanft & Backer, 
2011). This is the shift that entails and requires changes in the process of examination 
processes; the latter should build on the competencies that are to be acquired by students 
(Zawacki-Richter et ah, 2011). The movement from processes to outcomes in the higher 
education domains has already caused a series of shifts in course development and 
structure, but the issue of delivering competency-based assessments, especially in 
distance education, remains unresolved (Zawacki-Richter et ah, 2011). In the future, 
people will hardly choose jobs for life (Wright, Knight & Pomerlau, 1999). They will 
enter “portfolio” careers, meaning that the success of their career decisions will depend 
upon the evidence they provide on the wide array of skills, abilities and knowledge they 
have to fulfill the desired job (Wright et ah, 1999). This, in turn, may increase the use of 
portfolios as a form of knowledge assessment in higher education. By utilizing portfolios, 
this provides a visible and tangible representation of the individual's knowledge, skill, 
and dispositions. 

Higher education professionals and curriculum designers have continually relied 
on a complex mixture of assessment models, which include but are not limited to 
presentations, written assignments, presentations and tests. The use of portfolios in 
distance education has been rather scarce and, even then, ineffective and insufficient 
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(Zawacki-Richter et al., 2011). Reasons why higher education institutions have avoided 
using portfolios as an instrument of assessment are difficult to define. On one hand, the 
current state of research on the use of portfolios is focused on how useful it is for learners 
to use portfolio assessments (Tosh, Light, Fleming & Haywood, 2005). As a result, 
student perceptions of portfolio use in higher education are beyond the scope of 
contemporary research (Tosh et al., 2005). On the other hand, many teachers still expect 
that their institutions will provide training required to successfully implement portfolios; 
the situation is particularly difficult with distance teaching (Kim & Bonk, 2006). 

In terms of electronic portfolios that exemplify a novel approach to knowledge 
assessment in higher education, there has been little deeper reform (Lazerson, Wagener & 
Shumanis, 2000). Nonetheless, the rapid advancement of technologies necessitates the 
implementation of new assessment forms in distance education, but which forms of 
assessment are the most suitable for graphic design courses and how to implement them 
properly should be further clarified. Meanwhile, higher education institutions continue 
experiencing the pressure of the growing demand for higher education services, whereas 
the emergence of knowledge-based economy calls for reconsidering and adapting the 
purposes and processes of knowledge assessments to meet the needs of the new, global 
learning reality. In the age of competition, only those higher education institutions that 
constantly update their assessment procedures will have a chance to remain competitive 
and profitable in the long run. 

The need to explore the relevance of portfolio assessments in an online graphic 
design course is justified by; (a) the lack of attention to competency-based assessments; 
(b) the peculiar features of distance education that affect the nature of assessment 
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procedures; (c) the growing demand for learning in knowledge-based economy; and (d) 
the growing utilization of portfolio-based assessments in traditional higher education 
environments and their relationship to online courses. As previously mentioned, the 
paradigm shift from input towards outcome orientation justifies the development of new 
research projects that will explore the validity of portfolios as a form of outcome-based 
assessment procedure and its applicability in online graphic design courses (Zawacki- 
Richter et ah, 2011). Although the paradigm shift toward the outcome-based nature of 
higher education has been widely recognized, the potential benefits of portfolios as an 
instrument of outcome-based learning assessment are persistently disregarded (Zawacki- 
Richter et ah, 2011). 

Very often, portfolios become merely a “one more thing to do” and may not be 
treated as interesting and important by students who are already overloaded with 
assessments and tasks (Stone, 1998). Furthermore, portfolios have the potential to 
respond to the most recent trends in higher education, such as documenting authentic 
learning experiences, enhancing accountability in distance education, and helping 
colleges and universities to connect their students to multiple media and letting them 
express their experiences through YouTube and other technological innovations (Reese & 
Levy, 2009). While the demand for learning, especially distance learning, has been 
growing, technical advances challenge previous assumptions about higher education and 
assessment (Hanna, 1998). As a result, earlier methods of assessing knowledge may 
appear to be irrelevant in new global contexts. Moreover, the existing portfolio systems 
may fall short of their potential and undermine the effectiveness of new, technology- 
based approaches to higher education (Love & Cooper, 2004). 
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Certainly, higher education institutions have been relatively successful in the 
development of online portfolio assessment programs; unfortunately, the “intention of 
these online portfolio assessment programs has been to combine the benefits of 
traditional portfolio-based assessment with the paper saving and other benefits of online 
environments” (Love & Cooper, 2004, p.66). In other words, by implementing traditional 
portfolio assessments with online universities, colleges and other higher education 
facilities have created more problems than they can solve. First, traditional paper-based 
portfolio assessments implemented online, lack full integration with the course contents 
and design (Love & Cooper). Second, the heuristics and design criteria that underlie the 
creation of online education systems differ dramatically from those underpinning 
traditional education discourses (Love & Cooper). The specific features of distance 
education and their implications for portfolio assessment should not be ignored: distance 
education and learning by nature is associated with the concepts of isolation and the 
absence of face-to-face interactions with instructors (Suen, 1996). Apparently, there is an 
urgent need to explore the use and usability of portfolio assessments in distance 
education courses, to provide recommendations and improvements for education 
professionals. 

The following was the analysis of portfolio assessments and their effects on 
students in online graphic design courses. The crucial stages of the project will include a 
brief analysis of the issue in question and its background and history, problem statement 
and purpose of the study, its basic aims and objectives, as well as the discussion of 
methodology and the relevance of the study in the context of contemporary distance 
education and its relation to social change. All these stages were designed to create a 
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picture of completeness in the discussion of portfolio assessments and their applicability 
in distance education environments. It goes without saying that technological 
advancements trigger new developments in the field of education, but the main question 
to be answered is how to apply these developments properly. The goal of higher 
education, as well as education in general, is to deliver knowledge in ways that enhance 
students’ self-efficacy and comprehension, regardless of the course they choose. Graphic 
design courses are no exception: comprehension and self-efficacy are the foundational 
ingredients of effective education. Thus, how exactly do portfolio assessments impact 
students in the graphic design course? This was the question answered in this study. 

Background of the Study 

Portfolio assessments exemplify one of the most interesting and controversial 
aspects of higher education research. Much has been written and said about the nature of 
portfolio assessments and their applicability in higher education. Yet, despite the growing 
body of literature, the meaning of portfolios as an instrument of evaluation remains 
unclear. The growing number of portfolio definitions has made the task of describing 
portfolios in traditional and online education extremely problematic. Nevertheless, based 
on a thorough review of the current literature, the most accurate definition of a portfolio 
is: 

A specific collection of materials and documents, with the purpose of 
documenting a specific range of performance over a period of time. The portfolio 
serves as a component of self-evaluation and provides participants and outsiders 
with an overview of personal learning success, goals and results of the projects. 
(Zawacki-Richter et ah, 2011, p.45) 
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In other words, portfolios are collections of materials that are created by students or 
professionals with the goal of delineating the key processes and happenings in their 
career/leaming (Challis, 1999). Portfolios can also be defined as a collection of 
documents/evidence that substantiate the achievements, skills, competencies and learning 
accomplished by students within a given period of time (Love & Cooper, 2004). This 
definition also implies that portfolios encompass six different subcategories: (1) skills 
students intend to develop; (2) learning outcomes students are to achieve; (3) strategies 
used to enhance learning; (4) perfonnance indicators used to measure students’ progress 
and confirm they have achieved the desired outcomes; (5) evidence demonstrating the 
discussed performance indicators have been met; and (6) the quality of organization and 
presentation, so that the relationship among evidence and performance indicators is easily 
understood (Love & Cooper, 2004). 

For the purpose of this research, the difference between portfolios and e-portfolios 
should be clarified. As with portfolios, e-portfolios have been defined in more than one 
way. Reese and Levy (2009) offer a broad description of an e-portfolio, as a digitized 
collection of learning and experiential artifacts and accomplishments that represent 
changes in individual learning performance or that of a group and even institution. 
However, this definition fails to capture the intangible meanings underlying the essence 
and utilization of e-portfolios in higher education. Therefore, the definition provided by 
Joyes, Gray and Hartnell- Young (2010) served as the foundation for the development and 
implementation of this research project: 

An e-portfolio is the product, created by the learner, a collection of digital 

artifacts articulating experiences, achievements and learning. Behind any product, 
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or presentation, lie rich and complex processes of planning, synthesizing, sharing, 
discussing, reflecting, giving, receiving and responding to feedback. These 
processes - referred to here as ‘e-portfolio-based learning’ - are the focus of 
increasing attention, since the process of learning can be as important as the end 
product, (p.16) 

From the current literature, this definition will serve as the fullest and most recent. 
However, it is interesting that the discussed definition has not taken the paradigm shift 
from the process- to outcome - oriented learning for granted. Rather, the discussed 
definition of e-portfolios claims that the process of learning and its products are equally 
essential (Joyes et ah, 2010). This, however, does not change the essence of e-portfolios; 
nor does it reduce the validity of this study. On the contrary, this definition once again 
confirmed the broad applicability of e-portfolios as an instrument of assessing knowledge 
during and after the process of learning. Based on this definition, e-portfolios differ from 
traditional portfolios in that the fonner are created and delivered via digital environments 
and involve the use of the latest technologies. Moreover, it is possible to assume that the 
inclusion of technologies in higher education assessments and the inherently digitized 
nature of e-portfolios have greater potential to increase user engagement and 
collaboration among students and between them and their teachers/ facilitators in higher 
education environments (Attwell, 2007). 

It should be noted that portfolios have been found in all fields of professional 
development and higher education. They have been used for assessment and learning, 
promotion and appraisal (Klenowski, Askew & Camell, 2006). Surprisingly, there is no 
single standard application for portfolios in higher education (Meeus, Petegem & Looy, 
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2006), but in many cases, portfolios have served the purpose of developing reflective and 
teaching skills from pre-serving to postgraduate levels of learning (Klenowski et ah, 
2006). Portfolios can also be used for the purpose of strengthening the validity of 
summative evaluations (Klenowski et ah, 2006). The body of research concerning the use 
of portfolios in fonnative assessments at postgraduate levels is very scarce (Klenowski et 
ah, 2006). Nevertheless, it is still possible to classify higher education portfolios by the 
mode of implementation and purpose of use. It appears that in higher education, 
portfolios are often used for admission purposes, since they allow assessing student 
competencies and their readiness to enroll in higher education (Meeus et ah, 2006). 

During higher education courses, portfolio assessments can used to supervise and assess 
student competencies (Meeus et ah, 2006). Portfolios are also used upon entrance into the 
profession and during one’s professional evolution, as an instrument of ongoing 
professional assessment in the workplace (Meeus et al, 2006). The most common types of 
portfolios include smart and documentation portfolios, learner portfolios, introductory 
and course portfolios, meta-portfolios, and demonstration and instruction portfolios 
(Meeus et al., 2006). Although Meeus et al. (2006) cite these examples and types of 
portfolios, they also recognize that the current state of literature provides at least 28 types 
of portfolio clarifications. Therefore, defining and explaining these categories would be 
neither easy nor useful for this research. Since the goal of this study had nothing to do 
with the typology of portfolios and e-portfolios, this task would be successfully 
accomplished in future studies. 

The multitude of portfolio types and applications creates a picture of their 
universal applicability across all courses and learning models. However, this impression 
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was at least erroneous, since the use of portfolios as an instrument of assessment is 
conditioned by its nature, origins, and initial purpose (Meeus et al., 2006). Objectively, 
portfolio assessments reflect and fit in the atmosphere of co-constructivism in education 
that shifts the emphasis from individual responsibility for learning to a more collaborative 
view of the educational process (Klenowski et al., 2006). Portfolios are best used in the 
conditions where learners are given freedom to identify the most important aspects 
affecting the learning progress and act to make changes and improve their progress. 
Portfolios as a tool of assessment fit perfectly well in the environments where critical 
investigation, self-re flection, reorganization and reinterpretation of knowledge are crucial 
components (Klenowski et al., 2006). Portfolios facilitate the link between individual 
meanings and the contexts in which they are created (Klenowski et al., 2006). 

It is interesting to note that origins of portfolios as an instrument of assessing 
knowledge are found in the arts. Actually, the term ‘portfolio’ was borrowed from 
graphic arts to denote a collection of evidence substantiating the act of learning and 
measuring its progress (Snadden & Thomas, 1998). Today, portfolios can hardly be 
considered as new to the field of higher education, but for many years they have been a 
foundational pillar of graphic arts education, used to track the professional development 
of artists in long-tenn perspectives (Snadden & Thomas, 1998). It was not until recently 
that portfolios have come to be used beyond arts, for example, in medical education and 
science. Portfolios are becoming extremely popular as an instrument of measuring 
professional growth of employees in organizations as witnessed by Snadden & Thomas 
(1998). The assessment potential of portfolios can hardly be exaggerated: portfolios 
exemplify an extremely effective element of authentic assessment and allow measuring 
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changes in knowledge and personal and professional development over time from a 
practical, not theoretical, angle (Snadden & Thomas, 1998). 

The roots of portfolio assessment can also be traced to the times of John Dewey, 
who defended the value of self-reflection education and actually laid the groundwork for 
the development of portfolio philosophies in their current form (Snadden & Thomas, 
1998; Wang & Liao, 2008). Unfortunately, Dewey’s self-reflection ideas were not widely 
accepted, until Schon published his works on reflective practices and their value for 
higher education (Snadden & Thomas, 1998). Although portfolios are closely associated 
with the graphic arts, it was the Northern American system of education that brought the 
value of portfolios to the surface and gave rise to the new portfolio forms of assessment 
in schooling (Snadden & Thomas, 1998). Although, at first, portfolios worked merely as 
a collection of evidence supporting that the act of learning had occurred, they gradually 
changed to become an instrument of collecting and reflecting on students’ educational 
experiences (Snadden & Thomas, 1998; Wang & Liao, 2008; Zawacki-Richter, Hanft & 
Baecker, 2011). Even when portfolios became a popular fonn of summative assessment 
in American school education, their true value was persistently underestimated (Snadden 
& Thomas, 1998). The reliability and validity of the assessment results provided through 
portfolios were questioned (Snadden & Thomas, 1998). There was no explicit consensus 
among raters as for the role which portfolios could play in understanding student 
performance (Snadden & Thomas, 1998; Wang & Liao, 2008; Zawacki-Richter, Hanft & 
Baecker, 2011). Nevertheless, extensive use of portfolios in the classroom became a 
challenge that dramatically altered the classroom climate, the nature of interactions 
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among students and teachers and brought the novelty and creativity in the process of 
assessing students’ works (Rowe, 2008; Snadden & Thomas, 1998). 

Today, the use of portfolios raises a number of questions that refer to their 
applicability in highly diverse technological learning environments. Performance 
expectations in higher education increase (Mangin & Stoelinga, 2010), and new 
transformative models of learning give rise to electronic ways of assessing student 
knowledge, including e-portfolios (Stansberry & Kymes, 2007). More often than not, 
researchers explore the validity and assessment reliability of traditional and electronic 
portfolios (Barbera, 2009; Batson, 2002; J.O. Brown, 2001; Chenail, 2008; Conrad, 2008; 
Gulbahar & Tinmaz, 2006; McLoughlin & Luca, 2001; Strivens, 2007; Suen & Parkes, 
1996; Swan, Shen & Hiltz, 2006; Zawacki-Richter et ah, 2011). However, there has been 
no research on the impact of portfolios on students’ self-efficacy in distance education, 
and this research is intended to fill this gap. 

Statement of the Problem 

Although portfolios have become a popular instrument for assessing student 
learning in traditional education, their applicability in the context of distance education is 
poorly understood. The main reason why portfolios and distance education contexts lack 
an explicit connection is because distance education differs dramatically from the more 
traditional on-site classroom learning. Distance as the distinguishing feature of distance 
education has far-reaching implications for assessment and learning processes (Zawacki- 
Richter et ah, 2011). Distance also necessitates the development of new technologies and 
media that, in turn, distinguish distance education from all other forms of learning 
(Zawacki-Richter et ah, 2011). Distance education by nature implies the sense of 
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isolation in students, the lack of face-to-face interactions and delayed instructor feedback 
(Zawacki-Richter et al., 2011). Apparently, not all students feel satisfied with the absence 
of direct interactions with the instructor, and even the presence of sophisticated 
technologies cannot reduce this relationship gap. Nevertheless, the number of students 
taking online courses rapidly increases; the line between traditional on-campus and 
distance students is fading (Venable, 2010). Even those students who consider 
themselves traditional come to consider an opportunity to engage in an online course 
(Venable, 2010). More and more l st -year college students report having completed at 
least one online course even before they have been enrolled in college (Venable, 2010). 
Some universities and higher education departments require that students engage in at 
least one online learning course during their education process (Venable, 2010). This, in 
turn, raises the question of assessment quality and the role in which portfolios can play in 
enhancing students’ self-efficacy in distance education. 

Students are reported to experience considerable emotional and learning problems 
in online courses. The main problems that cause frustration in students include (a) the 
lack of immediate feedback; (b) the ambiguity of instructions and learning criteria 
provided on the Web; and (c) technical problems experienced by students in online 
courses (Hara & King, 1999). These emotional problems, in turn, undennine educational 
opportunities provided on the web (Hara & King, 1999). Moreover, distance learning 
shifts the responsibility for the quality of education and interactions to the learner, who is 
directly responsible for compensating for the lack of contact with other learners and the 
instructor (Suen, 1996). The active learner is the necessary precondition for making 
distance learning successful, productive, and cost-effective. Distance learners usually 
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tend to have clearer goals and reasons for engaging in a distance course, but the degree of 
responsibility for managing contacts with the instructor varies across individuals (Suen, 
1996). Apparently, students engaged in distance education need additional 
encouragement and support to achieve the desired learning outcomes. This is why the 
role of portfolio assessments in managing students’ self-efficacy in distance learning 
requires a detailed examination. 

Portfolios and e-portfolios exemplify a unique source of learning potentials, 
which motivate students to engage in reflective thinking and deep criticism of the 
learning progress (Barbera, 2009). In present day learning environments, the goal of 
education is to motivate students to construct new knowledge between them and their 
instructors, as well as between students themselves (Barbera, 2009). In this context, 
assessment procedures must be explicit and timely, to encourage and support students in 
achieving their learning goals. The problem is that assessment is no longer an instrument 
of grading but is a wide concept or philosophy of motivating students to move along the 
learning continuum. The problem is whether or not portfolios should be used to assess 
knowledge in distance design courses and how portfolio assessments can affect students’ 
self-efficacy and comprehension in an online graphic design course. 

Purpose of the Study 

The purpose of this comparative case study was to explore the impact of portfolio 
assessment on students’ perceptions of self-efficacy and comprehension in a graphic 
design course. Reasons why the purpose of the study was justified are numerous. 

Through discovery and research, the current state of literature confirms that assessment 
tools used by on-ground institutions differ greatly from those used by online institutions 
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in the higher education domain. The body of literature covering portfolio assessment and 
their usability as a fonnative assessment tool in student assignments highlighted the 
benefits offered to students and their effects on achievement (Chung, 2008; Kicken, 
Brand-Gruwel, Merrienboer & Slot, 2009; Smith & Tellema, 2007; Stansberry & Kymes, 
2007; Venable, 2010; Zawacki-Richter et ah, 2011). This study will focus on the analysis 
of portfolio assessment and their potential to enhance comprehension and students’ self- 
efficacy in an online graphic design course. 

The intent of the study was to address the existing gap in knowledge concerning 
the use of portfolio assessments in distance education and their implications for the 
quality of learning in online courses. The fact is that, with the rapid advancement of 
technologies and online education as a form of knowledge delivery, the need for 
developing new fonns of online assessment has been mostly unsatisfied. While 
instructors need better knowledge of assessment procedures and their online applicability, 
students need greater confidence to cope with the responsibility and processes provided 
in online education environments. The results of the study, on the one hand, will guide 
further research in the field of portfolio assessment in distance education and, on the 
other hand, provide recommendations to improve the quality of assessment procedures 
used in distance education. Whether or not the results of the study will be applicable in 
distance courses other than design is difficult to predict; eventually, portfolios by their 
nature are closely connected with design. Nevertheless, the study will lay the groundwork 
for evaluating the validity of portfolio assessments in distance education and their 
implications for students’ self-efficacy and comprehension. 
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Research Questions 


The goal of the study was to achieve an understanding as to whether or not 
portfolio assessments positively affect students’ self-efficacy and comprehension in an 
online graphics design course. By researching one online Digital Imaging course, allowed 
the researcher to analyze the possibility of using this study and research method within 
other online, graphic design courses such as; Illustration, Page Layout, Product 
Packaging and Digital Layout, just to name a few. In addition, the study focused on the 
analysis of student-instructor-assessment relations in an online graphic design course. 
Another question was whether or not it would be worth using portfolios for formative and 
summative assessment in distance education, particularly, in an online graphic design 
course. The principal variables explored in this study include assessment portfolios, and 
students’ perceptions of self-efficacy and students’ comprehension in an online graphic 
design course. As a result, the research questions of the study included: 

How do portfolios impact students in an online graphic design course? 

How do portfolios impact students’ perceptions of self-efficacy? 

How do portfolios impact students’ comprehension? 

Theoretical Framework 

The current state of research into portfolios and their use in online education 
suggests that the analysis of portfolio use in distance learning warrants the application of 
complex theoretical frameworks. This being said, Mezirow’s theory of transformation is 
believed to exemplify the most relevant approach to the study. Mezirow’s theory of 
transformation is a constructivist theory that focuses on the “cognitive restructuring and 
integration of experience, action and reflection” (Stransberry & Kymes, 2007, p.489). 
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Mezirow’s theory of transformation is a constructivist theory of adult learning, and this is 
the main reason why this theory was used as the foundation for this research project. 

According to Mezirow (1997), transfonnative learning is the process that affects 
change by means of the frame of reference. Adults develop and acquire a coherent body 
of experiences which serve the main frame of reference while defining the meaning of the 
world (Mezirow). In other words, frames of reference are the experiential lens through 
which adults develop their vision of the surrounding reality. They produce selective 
impacts on adult feelings, perceptions and actions, leading to the rejection of ideas that do 
not reflect adult preconceptions about the world and acceptance of those ideas that allow 
adults to realize their mental and behavioral intentions for their own benefit (Mezirow). 
Based on Mezirow’s theory of transformative learning, adults use critical reflection to 
transform their frames of reference (Mezirow,). Adults tend to become critically 
reflective of the assumptions made by other people or generated under the influence of 
changes in political, social, economic or cultural environments (Mezirow). 
Consequentially, self-reflection is the fundamental driver of personal transformations. 
Here, the link between self-reflection, personal transfonnation, and the use of assessment 
portfolios in distance education should be explained. Since learning is intended to cause 
positive transformations in learners’ worldview, and these transformations, according to 
Mezirow (1997) take place through critical self-reflection, then portfolios fit perfectly 
well in the contemporary conditions of knowledge delivery in higher education. 
Assessment portfolios can become that very means of self-reflection that drives personal 
transformation and expands students’ worldview. However, this assumption warranted 
further analysis and validation, and this was actually the goal of this study. 
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According to Mezirow (1997), there are four types of learning processes. First, 
adults can seek to support their viewpoint, by finding and using available evidence that 
supports their view (Mezirow, 1997). This evidence will eventually reinforce the existing 
ethnocentric biases and strengthen the intensity of the point of view (Mezirow, 1997). 
Second, adults learn by seeking new points of view (Mezirow, 1997). “We can encounter 
a new group and create new negative meaning schemes for them by focusing on their 
perceived shortcomings, as dictated by our propensity for ethnocentricity (Mezirow, 

1997, p.7). Third, adults can leam to transform their viewpoints and develop new 
knowledge of cultures in ways that transform the existing viewpoint and eliminate the 
existing misconceptions (Mezirow, 1997). Fourth, although rare, adults can transfonn 
their ethnocentric biases and attitudes toward the reality in which they live. However, 
these processes are very rare and demand extreme efforts and willingness to transform 
personal attitudes and beliefs. 

Mezirow ’s theory of transformative learning has far-reaching implications for 
understanding the process of distance education, the role and place of portfolio 
assessments, as well as the effects which portfolios might cause on students’ self-efficacy 
and content comprehension. To begin with, recognizing the function of learning 
transformation and transformative learning in the context of distance education 
contributes to a better understanding of adults’ learning needs and, consequentially, 
makes the role and function of adult education more explicit (Mezirow, 1981). 
Furthermore, adult educators must understand that learning is a multifaceted process that 
takes many forms; as a result, the analysis of the relationship between learning and 
assessment can become extremely complicated. Finally, to ensure the efficacy of adult 
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learning, instructors must assist learners in getting aware and critical of their own and 
other’s preconceptions, beliefs and assumptions (Mezirow, 1997). Adult learners 
definitely need to discover and understand their frames of reference and redefine the 
obstacles and problems they encounter in their lives, through transfonnative learning and 
self-reflection (Mezirow, 1997). In this sense, portfolios as a form of assessment hold a 
promise to facilitate critical self-reflection among adult learners and, therefore, become a 
reliable driver of transfonnations in adults. Again, these assumptions need further 
validation and analysis. This is why Mezirow’s transfonnative learning theory was used 
in this research and this is also why this research project is being implemented. 

Vygotsky’s theory will add relevance to Mezirow’s theoretical framework. Based 
on Vygotsky’s (1978) theory of learning, “the mind is not a complex network of general 
capabilities such as observation, attention, memory, judgment, and so forth, but a set of 
specific capabilities” (p.31). Vygotsky (1978) suggests that each of these specific 
capabilities is independent of others and, moreover, develops independently. For 
Vygotsky (1978), learning is more than the development of thinking abilities. The 
essence of learning is in acquiring numerous thinking capabilities about many different 
things. Learning does not change individuals’ focus and attention but teaches them to 
expand their worldview and focus on more than one thing (Vygotsky, 1978). Therefore, 
any improvement caused by learning in one capability will affect other abilities, only to 
the extent that they are related (Vygotsky, 1978). Vygotsky’s theory implies that portfolio 
assessments can measure changes in individual capabilities as a result of learning but can 
hardly measure their relations, as far as these capabilities are believed to be unrelated. 
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Simultaneously, it is with the help of portfolio assessments that changes in various 
capabilities under the influence of learning can be measured and evaluated. 

Nature of the Study 

The current state of research demonstrates the utility of qualitative methodologies 
as instruments of analysis for researchers. According to Stansberry and Kymes (2007), 
the importance of qualitative frameworks in the study of portfolios cannot be disregarded. 
The qualitative study conducted by Stansberry and Kymes (2007) had indicated that 
students are actively involved with learning, and portfolios strengthen the feeling of joy 
and satisfaction from being engaged in the process of education. Creswell (2009) stated 
“qualitative research is a type of educational research in which the researcher relies on 
the views of participants” (p.46). This was the main methodological framework used as 
the foundation for this research study. 

The study sample consisted of 20 participants in a graphics design course, 
selected using random sampling procedures. Creswell (2008) justified the relevance of 
random sampling as a method to randomly assign individuals to the study group. In doing 
so, provides each student the same probability to be included within the research project. 
In this study, random sampling defined as the process of systematic selection of subjects 
for analysis, without having taken local and contextual features into account (Cohen, 
2006). The course lasted five weeks and, once in a week within their course (any week), 
students were required to create a portfolio of the finished work to be assessed by the 
researcher. Their work would be a specific design project for the given week. Students 
were also expected to complete an open-ended survey, which were then analyzed with the 
help of qualitative research techniques. At the end of the course, students’ self-efficacy 
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and its relation to the use of portfolio assessments during the course were evaluated 
through the data collected via an open-ended survey distributed among the participants. 
Comprehension levels were measured by means of rubrics that measured the 
comprehension of the required design skills, as well as the analysis of design concepts. 
Eventually, the data gathered on comprehension and self-efficacy was combined to create 
a complete picture of the relationship between self-efficacy, comprehension and 
portfolios in an online graphic design course. 

Significance of the Study 

The study proved to be of particular relevance for educators, especially those 
working in the field of distance learning. Since the time when computers have made 
virtual classrooms possible (Suen, 1996), the number of education professionals working 
in the distance education domain has continued to increase. The advantage of distance 
education is in that students do not need to be physically present in the classroom and 
participate in the course activities at one and the same time (Suen, 1996). This, however, 
has placed new demands on educators, who lose the sense of control over students’ 
activities and, therefore, cannot adequately evaluate their learning progress. The results of 
this study will provided an understanding as to whether or not portfolios as a form of 
formal assessment enhance students’ self-efficacy and comprehension and, 
consequentially, should be used in online graphic design courses. 

The study was extremely important for scholars and researchers in the field of 
online education. The number of studies about and the knowledge of distance education 
and its philosophies have been constantly increasing. Nevertheless, several questions 
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concerning the use of portfolios in distance education continue to persist: (1) how to 
resolve the conflict between individual learning and large-scale learning priorities, such 
as the development of a new professional group or the creation of a solid foundation for 
lifelong learning; (2) how to assess student work that are not and cannot be standardized; 
(3) what criteria should be used to consider the influence of students’ feelings, 
perceptions, feelings and attitudes on assessment results; and (4) how to deal with the 
issues of privacy while using portfolios (Snadden & Thomas, 1998). All these questions, 
although not directly related to the purpose of this study, were inherently related to the 
use of portfolios in distance education. Researchers working in the field of distance 
learning and exploring various models of assessment and their applicability in present 
day education will be able to use the results of this study as a guiding light in their future 
research endeavors. It is possible that the results of the study will also give rise to the 
emergence of new portfolio assessment types, which will have made this study 
potentially significant for both scholars and practitioners. 

The impact of the study results may cause on practitioners should not be 
disregarded. The past 25 years have witnessed an influx of adult learners into the higher 
education institutions within the United States (J.O. Brown, 2002). As a result, they bring 
a wealth of lived experiences and share them with their supervisors and education 
practitioners (J.O. Brown, 2002; Zawacki-Richter et ah, 2011). However, until present, 
little has been known on how exactly portfolios affect adult learners and what these 
learners can derive from the use of portfolios and e-portfolios in higher education (J.O. 
Brown, 2002). For higher education practitioners, this study will become a revelation 
about how confident and self-efficacious students may feel with portfolios. Although the 
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study is focused on the graphics design course, the results of this study will shed light on 
and create the basis for developing similar research projects in other learning settings. 

Despite its growing popularity, distance education has been a relatively new field 
of knowledge and practice. Consequentially, instructors in online courses either have 
lacked knowledge to implement new fonns of assessment or have had to adopt traditional 
portfolio forms to be used in online education. More often than not, the use of portfolios 
in distance education is either insufficient or scarce (Zawacki-Richter et ah, 2011). In this 
sense, the practical value of this research project can hardly be overstated. The results of 
the study will help to clarify the relationship between assessment portfolios, students’ 
perceptions of self-efficacy and their comprehension levels and, simultaneously, remove 
or resolve the fears contemporary educators may hold with regard to using portfolios in 
online graphic design courses. 

The study will close the existing gap in knowledge about portfolios and student 
self-efficacy. Most studies have evaluated the objective, not subjective, side of using 
portfolios for assessment purposes in education (Barbera, 2009; Batson, 2002; R.E. 
Brown, 2001; Chenail, 2008; Conrad, 2008; Gulbahar & Tinmaz, 2006; McLoughlin & 
Luca, 2001; Strivens, 2007; Suen & Parkes, 1996; Swan, Shen & Hiltz, 2006; Zawacki- 
Richter et ah, 2011). However, as the number of students engaged in online courses has 
increased, traditional forms of assessment pose a serious challenge to the quality of 
student outcomes. Since students experience serious emotional and cognitive problems 
due in part by the lack of face-to-face interactions, the relationship between portfolios 
and students’ feelings of self-efficacy have to be clarified. The results will add to the 
current state of practical knowledge about portfolios. 
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The results of the study can become a guide in the development of new policy 
recommendations for distance education, specifically in graphic design courses. Once the 
relationship between portfolios, self-efficacy and comprehension is established, the place 
of assessments and portfolios in distance learning can be clarified. “If we are to 
revolutionize and dramatically enhance education, it will require engaging students and 
getting them to think meaningfully and strategically about learning, especially the 
learning of core competencies” (Abrami et ah, 2008, para.l). However, revolutionizing 
higher education will be impossible without a solid empirical basis supporting the 
relevance of e-portfolio assessments and their positive effects on students’ self-efficacy. 
This study will help to clarify the effects that e-portfolios produce on adult learners, and 
guide further examinations in the field of portfolio assessments, as related to other, non- 
design courses. 

The study was directly related to the notion of social change, which has shown to 
be not possible without fundamental commitments to quality education, through the 
construction of knowledge, meanings, perspectives, and personal transformations 
(Mezirow, 1997; Meeus, Petegen & Looy (2006). Therefore, the quality of education is at 
the heart of all social change projects. The study ultimately provided with its results, its 
guidances, and recommendations to improve the quality of distance education, through 
the more appropriate use of portfolios in online graphic design courses. Consequentially, 
the results of this study will have contributed to positive social change, with the help of 
the knowledge obtained by students through critical self-reflection and transformative 
learning and, possibly, the use of portfolios in online graphic design courses (Meeus, 
Petegen & Looy, 2006; Mezirow, 1997). 
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Definition of Terms 


Assessment. This study defines assessment as the process of accumulating, 
analyzing, and systematizing infonnation about students’ learning results and their 
implications for change in individual intellectual capacity (Stasiunaitiene & 
Kaminskiene, 2009). 

Comprehension. This study defines comprehension as a student’s capacity to 
perceive and understand the meanings communicated by online instructors and the 
requirements placed on them in terms of graduation (Caldwell, 2008). 

Design course. This study defines design course as a brief online distance course 
of five weeks that qualifies students to complete the undergraduate degree of a Graphic 
Designer. 

Distance education. This study defines distance education as a field of education 
that presupposes delivering knowledge and teaching content by technological means, to 
recipients that are not physically present in one place and at one time (Hanna, 1998). 

Distance learning. This study defines distance learning as that which occurs not 
in the actual presence of the teacher, either in the next building, at home, or in a place 
located hundreds and thousands of miles away (R.E. Brown, 2001). 

Evaluation. This study defines evaluation as the process of measuring the degree 
to which the learners have achieved the desired learning standard (Stasiunaitiene & 
Kaminskiene, 2009). 

Frame of reference. This study defines frame of reference as a coherent body of 
individual experiences, including concepts, associations, feelings, values, and conditions 


24 



responses, which are used to govern transformative learning in individuals (Merizow, 
1997). 

Learning achievements. This study defines learning achievements as skills, 
knowledge, abilities, and attitudes that a person (student) demonstrate in the course and 
after the process of learning takes place; learning achievements can be also considered as 
the results of subjective learning or objective studying results (Stasiunaitiene & 
Kaminskiene, 2009). 

Portfolio. This study defines a portfolio as being “a specific collection of material 
and documents, with the purpose of documenting a specific range of performance over a 
period of time. The portfolio serves as a component of self-evaluation and provides 
participants and outsiders with an overview of personal learning success” (Zawacki- 
Richter et al., 2011, p.45). 

Self-efficacy. This study defines self-efficacy as a “persons’ confidence in their 
abilities to complete tasks or reach goals but is not based entirely on actual experience 
while perfonning these tasks in the past” (DeTure, 2004, p.22). 

Transformative learning. This study defines transformative learning as the 
process of effective change through experiences, perceptions, preconceptions and other 
frames of reference (Mezirow, 1997). 

Assumptions 

Given that qualitative research was the foundational element of the present 
research project, the following assumptions dictated the process of research and analysis. 
To begin with, the researcher assumed that the reality was comprised of multiple 
meanings, most of which are subjective and need to be clarified. In this sense, students’ 
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words while describing their experiences with portfolios created the foundation for 
understanding the effects on self-efficacy and comprehension. Furthermore, it was 
initially assumed that teachers and students had the basic understanding of portfolios, 
their meaning, content, and outcomes. Students did not need to have experience using 
portfolios as a form of assessment but already had the knowledge of what portfolios 
entailed. However, they may have needed to understand the essence of portfolio 
assessments and their implications for the quality and efficiency of the learning process. 
This was why the participants had to be instructed about the essence and significance of 
portfolios in traditional and distance education. 

ft was also assumed that all participants entered the graphic design course 
voluntarily, and with having the goal of improving their design knowledge and skills. 

This was the basis for evaluating the levels of self-efficacy and changes during the course 
and with the help of portfolios. Finally, it was assumed that all students had a capability 
to articulate their thoughts and perceptions effectively. 

Limitations 

The proposed project design and methodology are not without limitations. Most 
limitations are associated with the deficiencies inherent in qualitative research. To begin 
with, the issue of transferability should not be disregarded, in qualitative research 
transferability is somewhat similar to generalizability in quantitative studies and refers to 
the applicability of the study results in contexts other than the study itself (Tappen, 2010). 
However, qualitative studies are not designed to ensure the generalizability of findings, as 
in case with quantitative studies (Tappen, 2010). The goal of qualitative research is not to 
generalize the findings but to perfonn an in-depth examination of a particular 
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phenomenon within a given context (Tappen, 2010). To ensure greater reliability of 
qualitative research results, they can be connected to other findings and studies, and 
careful comparisons across people and places will create a more complete picture of the 
phenomenon in question. 

Another problem was that the study was focused on the analysis of the 
relationships among students’ perceptions of self-efficacy, course comprehension, and 
the use of portfolios as a form of assessment in an online graphic design course. 
Qualitative researchers tend to focus their analysis on a small number of participants, and 
it is difficult to imagine that the 20 students involved in the present study would represent 
the entire student population engaged in an average graphic design course within distance 
education. 

Another problem was that the research project built on the use of the subjective 
data, which made it difficult to re-channel non-measurable responses and contexts into 
measurable data (Terry, 2005). According to Creswell (2008), qualitative validity implies 
that the “researcher checks for the accuracy of the findings by employing certain 
procedures” (p. 190). In the case of this study, all procedures were followed to provide 
possible themes through the use of coding tenninologies through open-ended surveys. 

The only limitation is the possibility of interpretation of the reader and coding keywords 
properly to meet all participant responses. As a result, Gibbs (2007) confirmed 
“qualitative reliability indicates that the researchers approach is consistent across 
different researchers and different projects” (as cited in Creswell, 2008, p. 190). 

Moreover, the issue of the researcher’s bias while interpreting subjective data 
should not have been easily dismissed. Qualitative studies are particularly susceptible to 
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the risks of interpretation bias (Creswell, 2008: Mezirow, 1997; Rezaei & Lovorn, 2010; 
Terry, 2005). Here, the most important is the quality of the relationship between the 
researcher and research subjects. The reliability of the researcher and the researcher’s 
ability to detect and reduce the effects of personal bias on the interpretation of study 
findings will play a crucial role for the quality of this research project. Qualitative 
researchers may become too closely involved with the research subjects, which will 
threaten the quality of the study findings (Terry, 2005). However, the lack of engagement 
with the research subjects, which is not regarded as desirable in quantitative studies, may 
be highly desirable in qualitative research projects (Bryman & Bell, 2007). Through 
greater involvement with participants’ lives, the researcher can better understand the 
world they are seeing through their eyes (Bryman & Bell, 2007). These may be typical 
limitations for an average qualitative study; however, these issues have also been 
addressed within this study. The researcher will only have contact with the participants 
within this study during the data collection period but only in the role of being the 
instructor for the assigned course(s). All “immediate” interaction will take place between 
the participants and the Research Assistant (RA) assigned for this study to keep from 
creating possible problematic scenarios. 

Eventually, the issue of subjects’ honesty had to be considered. Researchers can 
never be confident that the infonnation provided by the research participants was truthful. 
This is particularly the case of students’ perceptions of self-efficacy. “Social scientists, 
unlike lawyers, must often make decisions regarding the subjective realm without 
recourse to a process of closely interrogating or interviewing a subject” (Carspecken, 
1996, p.74). Moreover, social scientists must have an ability to distinguish truthful 
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information from the untruthful inferences made by students with regard to their 
perceptions of the online course. Students who feel that their revelations threaten their 
successes in online education may be reluctant to uncover the true feelings they held with 
regard to their self-efficacy and participation in an online graphic design course. These 
were the problems that could threaten the validity and reliability of the study results. For 
the purpose of this study, participants were informed that their instructor who was also 
the researcher would not have knowledge of who was or was not participating within the 
study. By assigning a Research Assistant, alleviated any of the possible fears of honesty 
within their surveys. 

Summary and Organization of the Remainder of the Study 

This section was devoted to the discussion of the study problem, purpose, and 
research questions. The study was intended to close the existing gap in literature and 
practical knowledge of portfolios and their applicability as a fonn of assessment in 
distance graphic design education. The main question to be answered was how portfolios 
impacted students engaged in distance graphic design courses. More specifically, the 
study was designed to understand whether or not portfolios affected students’ perceptions 
of self-efficacy and their comprehension of the content delivered online. Therefore, the 
purpose of this comparative case study was to determine how, and to what extent, 
portfolios impact students’ self-efficacy and impact their comprehension levels while 
completing the portfolio process. 

The growing popularity of distance education, the growing number of students 
engaged in online courses, and the difficulty applying traditional assessments in distance 
education create a context that justifies the importance of the study. The study has far- 
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reaching implications for educators, education researchers, and policymakers. The 
difficulties and limitations of the proposed qualitative design should not be disregarded. 
The researcher’s reliance on the subjective data, along with the limitations of 
participants’ readiness to reveal their perception of their self-efficacy may have posed a 
serious challenge to the validity and reliability of the study results. 

The remaining chapters for this study include: review of literature, methodology, 
data collection and analysis, results, conclusions and recommendations. Chapter 2 
includes a detailed review of literature. The goal of the review is to reconsider the most 
recent findings in the context of self-efficacy and comprehension in distance learning, 
identify the existing research gaps and possible ways to deal with them. Chapter 3 will 
cover methodology requirements and concerns, providing the explanation and 
justification for the research methods used in this dissertation. Chapter 4 will shed light 
on the instruments used to collect the data for this dissertation and describe the process of 
data analysis. In Chapter 5, the results of the research will be presented, followed by 
conclusions and recommendations. 
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CHAPTER 2. LITERATURE REVIEW 


Introduction 

Distance education has become part of the higher education realities. New 
assessment models are being developed to meet the growing demand for quality 
education and learning in highly technological environments. The shift from process- to 
outcome- oriented perfonnance measures necessitates the creation of assessment 
procedures that build on student competencies (Zawacki-Richter et ah, 2011). The entire 
field of higher education has already undergone a series of dramatic shifts, such as the 
move towards outcome-based orientation and the call for greater transparency and 
accountability in the field of learning (Reese & Levy, 2009; Zawacki-Richter et ah, 
2011). Yet, higher education institutions are still far from improving their assessment 
results. One of the main problems is in that professionals in distance education use 
different assessment methods and not all of them are appropriate and fit in the complex 
conditions of online knowledge delivery (Zawacki-Richter et ah, 2011). One of the 
possible reasons why portfolios are still scarce in distance education is because education 
professionals lack an insight into the utility and usability of these assessment procedures. 

The body of research into the role of portfolio assessments in e-learning 
environments constantly increases. While higher education institutions experience the 
growing demand for learning and education services, adopting unique and effective 
models of education can become a serious competitive advantage. One of the main goals 
of this research is to close the existing gap in literature and link the use of portfolio 
assessment models in distance learning to self-efficacy and comprehension. 
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Consequentially, this research would have been impossible and incomplete without 
reviewing relevant research that has been written and said about portfolio assessments, 
their role and history, as well as the importance of qualitative research designs in the past 
5 years (2007-2012). The importance of this research and the review of literature are 
further justified by (a) the overall lack of attention to competency-based assessments in 
learning; (b) the overall peculiarity of distance education compared to classroom-based 
learning; and (c) the growing popularity of reflective portfolio assessments in classroom- 
based higher education. 

One of the main goals of this review is to justify the use of qualitative 
methodology in the analysis of portfolio assessments and their effects on students’ self- 
efficacy and comprehension in an online graphic course. A detailed review of literature 
will create the conditions needed to ensure comparability and enhance the interpretive 
characteristics of the research results. Eventually, although literature review alone cannot 
solve the problem of using portfolio assessments in distance education, its results can 
frame the basis for obtaining and delivering quality research results. 

Portfolios and e-portfolios can become a unique source of learning achievements, 
which can motivate students to achieve new learning horizons. According to Barbera 
(2009), the goal of education is to enable learners to construct knowledge between 
students, as well as between students and instructors. The body of literature covering 
portfolio assessment and their usability as a fonnative assessment tool in student 
assignments highlights the benefits offered to students and their effects on achievement 
(Chung, 2008; Kicken, Brand-Gruwel, Merrienboer & Slot, 2009; Smith & Tellema, 
2007; Stansberry & Kymes, 2007; Venable, 2010; Zawacki-Richter et al., 2011). 
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However, it is never too late to emphasize the benefits and potential weaknesses of 
portfolio use in education, based on previous experiences. Earlier findings and previous 
experiences can become a unique source of primary knowledge and help to avoid the 
common fallacies inherent in professional research. This literature review will include the 
following elements: qualitative studies related to portfolio assessments, self-efficacy and 
comprehension; portfolios as assessment tools, and comprehension. 

Qualitative Comparative Case Study 

Qualitative designs remain one of the predominant design frameworks in the 
study of portfolio assessments. Reasons why qualitative designs are so popular in the 
study of portfolio assessments are numerous and diverse. On the one hand, qualitative 
research fits in the meaning and nature of portfolio assessments, which rely on 
storytelling and narration. On the other hand, the study of portfolio assessments demands 
greater attention to the exploratory and interpretive aspects of portfolio assessment use 
and its implications for social behaviors in education. Using qualitative study designs in 
the study of portfolio assessments is not an easy task. The pedagogy of portfolio 
assessments is extremely complicated. However, the current state of research suggests 
that, in the analysis of portfolio assessments in distance learning, there is no study design 
better than qualitative case study. The goal of this section is to highlight the rationale 
behind the use of qualitative study designs in this study and identify possible 
misconceptions. 

Qualitative studies remain one of the most popular elements in the study of 
portfolio assessments, self-efficacy and comprehension. The current state of research 
provides solid arguments to justify the implementation of qualitative research designs in 
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the analysis of portfolio assessments in various learning settings. This section begins with 
an overview of qualitative studies in portfolio assessment and proceeds to the analysis of 
qualitative research in the study of self-efficacy and comprehension. The latter appears to 
be the most problematic and, simultaneously, the most interesting object of qualitative 
research. The findings of this literature review will facilitate the development and 
implementation of qualitative research frameworks for the analysis of portfolio 
assessments and their effects on learners’ self-efficacy and comprehension in an online 
course. 

Qualitative case study and portfolio assessments. Present day research provides 
a wealth of infonnation regarding the use of qualitative research design in the analysis of 
portfolio assessments and their effects on learning. The appropriateness of qualitative 
designs and case studies in the analysis of portfolio assessments should not be 
disregarded. Different researchers point to different aspects of qualitative design in the 
study of portfolios as a form of assessment in education and learning. Papinczak, Young 
and Groves (2007) suggest the presence of a strong epistemological grounding in the use 
of qualitative case studies: qualitative analysis is not intrusive and invites individuals to 
engage in exploratory conversations, which are parts of both the hermeneutic and 
pedagogical processes. Qualitative methodologies enable researchers to discover the 
meanings adults tend to endow with their learning experiences (Papinczak et ah, 2007). 
By contrast, quantitative studies, such as quantitative questionnaires and surveys, can 
restrict respondents to the questions deemed important by the researcher (Grant, 

Vennunt, Kinnesrsley & Houston, 2007). Within quantitative methodologies, respondents 
simply have no freedom of self-expression and self-exploration (Grant et al., 2007). 
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Despite the growing popularity of qualitative designs and their utility in the analysis of 
portfolios and e-portfolios, the gap in qualitative research continues to persist. Reardon 
and Hartley (2007) write that a wealth of research informs stakeholders about the role of 
portfolio in assessment, criteria for using portfolios, characteristics and qualities of good 
portfolios which, in the absence of the qualitative angle, fails to integrate portfolios with 
the sequence of learning experiences leading students towards the desired goal. It is no 
wonder that the study of portfolio assessments calls for the use of qualitative research 
designs. 

In the analysis of portfolios as an instrument of assessment, qualitative case 
studies and similar qualitative methods are believed to be more appropriate than 
quantitative frameworks. Plaza et al. (2007) claim that, bearing in mind the inherently 
qualitative nature of reflective portfolios, qualitative methods may be more appropriate 
than quantitative ones. Chau and Cheng (2010) further support this opinion, by saying 
that, “understanding the e-portfolio experiences of participants and seeking to identify 
ways in which such experiences are understood and perceived by participants” mandates 
the use of qualitative approaches (p.935). The reason is that qualitative research 
paradigms substantiate and emphasize the intricacy of human realities expressed in 
assessment portfolios (Chau & Cheng, 2010). Qualitative comparisons are of particular 
use in contexts where the main themes in participants’ reflections have to be examined 
(Lo, 2010). Since many participants’ responses are loosely structured, statistical analyses 
will fall short of resources and capabilities to produce relevant study results (Lo, 2010). 
Finally, portfolios’ narrative nature and the emphasis on storytelling justify and require 
using qualitative studies to explore their themes and aspects (Fisch, 2010). It is through 
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qualitative frameworks that an in-depth view of students’ experiences in distance 
education and portfolio assessment can be developed (Chetcuti, 2007). 

Qualitative designs have proved to be extremely relevant in the analysis of 
portfolio assessments, since portfolios are closely associated with the concept of 
competence, which encompasses knowledge, communication, reasoning and technical 
skills, and cannot be always measured in quantitative terms (Cox & Irby, 2007). Both 
portfolios implementations and competence development are contextual and require the 
use of sophisticated qualitative solutions (Cox & Irby, 2007). Qualitative analysis meets 
the criteria of comparability, fairness, cognitive complexity and authenticity, all of which 
are also characteristic of portfolio assessments (Baartman et ah, 2007). For example, 
Tochel et al. (2009) used qualitative research to identify the main themes in the 
effectiveness of portfolios in postgraduate education. Ntuli, Keengwe and Kyei-Blankson 
(2009) applied to qualitative methods to analyze teacher perceptions’ of electronic 
portfolios. Apparently, qualitative methods hold a promise to facilitate the analysis of 
portfolio assessments in higher education, although self-efficacy and comprehension 
could benefit from the use of qualitative designs. 

Qualitative study and self-efficacy. Like portfolio use in general, self-efficacy 
has proved to be a unique object of qualitative analysis. Unfortunately, not all researchers 
realize the value carried by qualitative designs. This is probably why mixed studies 
continue to dominate the current state of self-efficacy research. Kaskaya, Unlu, Akar and 
Sagirli (2011) used mixed study designs to explore the effects of themed movies on 
teachers’ perceived self-efficacy and found that, at the end of the intervention, students’ 
perceptions of the teaching profession improved considerably. Poellhuber, Chomienne 
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and Karsenti (2008) further analyzed the effects of peer collaboration and collaborative 
learning on learners’ self-efficacy in distance courses and discovered that peer 
interactions had the potential to improve students’ self-efficacy. Likewise, Gunawardena 
et al. (2010) applied to mixed methods of research and tried to predict learner satisfaction 
in an online education program. These findings have far-reaching implications for 
understanding the meaning of self-efficacy in online learning. On the one hand, self- 
efficacy is being influenced by a variety of factors, including themed movies, peer 
interactions and learner satisfaction (Gunawardena et al., 2010; Kaskaya et al., 2011; 
Poellhuber et al., 2008). On the other hand, given the popularity of mixed research 
designs, it is possible to assume that the concept of self-efficacy requires detailed 
statistical analyses, and mixed methods of research can ensure greater validity and 
reliability of qualitative findings, through triangulation. The most interesting, however, is 
the fact that self-efficacy, especially in online education, is often taken for granted. In 
other words, researchers in distance education fail to define and denote the meaning of 
self-efficacy. Whether or not the meaning of self-efficacy in distance education differs 
from that in classroom settings is difficult to define, but it is clear that researchers are 
becoming increasingly concerned about the applicability of qualitative methodologies in 
the study of self-efficacy and its implications for distance education. 

Saade and Kira (2009) perfonned a qualitative analysis of computer anxiety in e- 
leaming. In their study, self-efficacy was defined in terms of Bandura’s theory and 
denotes learners’ ability to judge how well they can executive a learning task to achieve 
the desired goal (Saade & Kira, 2009). By contrast, Gunawardena et al. (2011) suggest 
that, in online environments, self-efficacy is essentially about learners’ confidence in 
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utilizing technology to engage in productive learning. Apparently, both concepts require 
further validation and qualitative analysis, to ensure their applicability in distance 
learning. Although self-efficacy can be successfully measured by means of self-report 
forms and Likert-type scales (Bell & Kozlowski, 2008), only qualitative methods can 
expose the hidden meanings of self-efficacy in technological educational environments. 
Nonetheless, researchers keep using quantitative elements of research design in an 
attempt to look deeper into the nature and implications of portfolio assessments in 
various learning environments (Lee & Hwang, 2007). 

The choice of qualitative methods of self-efficacy analysis is closely associated 
with the fact that students’ perceptions of self-efficacy vary greatly across individuals, 
confirming the multifaceted nature of the self-efficacy concept and its complex 
relationship to learning and socialization in education (Gardner, 2010). Self-efficacy is 
inseparable from the so-called vicarious experiences, also known as social comparisons 
(Hutchison-Green, Follman & Bodner, 2010). Stated plainly, students develop their self- 
efficacy perceptions, by comparing their personal abilities to the perceived abilities of 
other students, and these experiences play a fundamental role in the fonnation of 
learners’ efficacy beliefs (Hutchison-Green et ah, 2010). Eventually, self-efficacy is a 
complex construct that is closely associated with the meaning and extent of learning 
motivation in students: according to Pajares, Valiante and Cheong (2007), self-efficacy is 
positively related to adaptive motivation, self-regulated learning, task goal orientation 
and perceived value of writing. All these constructs may have a measurable side but 
underline the value of qualitative, exploratory research frameworks that enable 
researchers to look beyond their complexity. 
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Qualitative studies and comprehension. As previously mentioned, qualitative 
research has become one of the central media of professional analysis in the study of 
portfolio assessments and their implications for self-efficacy. Likewise, qualitative 
approaches to the study of comprehension are becoming more prevalent. However, it 
should be noted, that the issues of measurability in the study of learning comprehension 
continue to persist. In this sense, the problems identified in the study of self-efficacy are 
very similar to those identified in the analysis of the contemporary comprehension 
research. Again, like in the study of self-efficacy, the meaning of comprehension is being 
taken for granted. Of all comprehension studies included in this review, only Brodie 
(2007) tried to discuss and evaluate the relationship between various levels of learning, 
including comprehension. Even then, Brodie’s (2007) definition of comprehension was 
borrowed from Bloom’s taxonomy of learning, where comprehension is explained as the 
process during which learners interpret, translate or summarize the information provided 
during the course and, consequentially, demonstrate their understanding of the learning 
event. Comprehension follows knowledge; the latter is limited to recognizing and 
recalling information and facts (Brodie, 2007). Definitely, comprehension is a higher 
level of learning which also involves interpretation and analysis (Brodie, 2007). Brodie’s 
(2007) qualitative analysis also suggests that comprehension lays the groundwork for 
applying information in situations different from the learning context. This is how 
comprehension transforms into application (Brodie, 2007). 

Unfortunately, the body of qualitative research into comprehension is very scarce. 
Over the past five years, few researchers tried to apply the benefits of qualitative designs 
to understand the meaning of comprehension. In most cases, comprehension is believed 
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to be a purely measurable construct, whose value is determined through scoring, grading, 
and other quantitative instruments. Only Rowe and Wood (2008) used qualitative 
analyses to understand student perceptions of feedback and their implications for 
comprehension. Suviniitty (2010) also used qualitative design to explore the complex 
relationship between lecturers’ questions and students’ perceptions of lecture 
comprehension. 

The sustained popularity of non-qualitative approaches in the analysis of 
comprehension reveals and confirms an ongoing concern regarding the importance of 
measurability in comprehension. The lack of trust to qualitative designs further 
strengthens professional commitment to quantification. For example, Rybarczyk et al. 
(2008) developed a quantitative research approach to explore the effects of case-based 
study designs on students’ comprehension of cellular respiration concepts. Yanguas 
(2009) and Suviniitty (2010) raised the question of measurability in the study of 
comprehension, and Murphy (2007) used mixed methodology to strengthen the validity 
of his qualitative findings. Unfortunately, comprehension as a concept lacks a qualitative 
insight. The latter, however, could explain the complexity underlying the process of 
comprehending new knowledge and information in distance and traditional classroom- 
based education. In light of the discussed research gap, this study looks like a unique 
attempt to understand what exactly stands behind the term “comprehension” and how 
students in design courses “comprehend” the new material. 

In summary, the current state of research offers a wealth of qualitative 
information regarding the nature of portfolio assessments, self-efficacy and 
comprehension. The most problematic is the qualitative analysis of comprehension, as the 
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latter is often regarded as a purely quantitative, measurable construct. Likewise, self- 
efficacy often becomes an object of quantitative and mixed studies. The lack of 
qualitative infonnation and understanding of these concepts, as well as the exploratory, 
reflective and narrative nature of portfolio assessments justify the implementation of 
qualitative designs for the study of portfolio assessments and their effects on students’ 
self-efficacy and comprehension in online design courses. 

Portfolios as a Type of Assessment 

Portfolios are hardly a new instrument of assessing learning. The history of 
portfolios as a form of assessment and their popularity in classroom-based learning 
suggest that portfolios exemplify a unique, complex, and extremely effective mechanism 
of assessing students’ progress towards their learning objectives. This, in turn, raises the 
question of appropriate portfolio use in higher education and their potential usability in 
distance education. This is why a detailed review of literature can expose the main 
benefits of portfolio assessments in different learning settings and identify possible uses 
of portfolios as a form of assessment. 

Portfolios are believed to have far-reaching implications for learners’ self-efficacy 
and comprehension. Here, special attention should be paid to the place which portfolios 
occupy in the contemporary visions of learning as a constructive activity. The 
relationship between portfolios, constructivism, self-efficacy and comprehension needs 
professional attention. Theories of Albert Bandura (1986), Jean Piaget (1967), Vygotsky 
(1978) and Mezirow (1981) should both create the basis and serve an analytical lens for 
the present study. All these theories used to be extremely popular among earlier 
researchers, especially in the self-efficacy and comprehension domains. The task of this 
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review is to link earlier findings to the study of portfolio assessments. Undoubtedly, 
portfolio assessments have a great future in distance education, but systemic knowledge 
of research findings can ensure that the assessment and learning potentials of portfolios 
are utilized to the fullest. 

History of portfolio assessments. The history of portfolio assessments is 
inseparable from the history of assessment, in general. The history of assessment in U.S. 
education dates back to the middle of the 19 th century, when assessment emerged in K-12 
education (Buzzetto-More, 2007). At that time, Horace Mann, a pioneer of assessment, 
used the first written examinations to measure learners’ progress in Massachusetts 
(Buzzetto-More, 2007). By the 1920s, a broad scientific movement emerged, leading to 
the development and implementation of large-scale testing which, by the middle of the 
1960s, transformed into the National Assessment of Education Progress (Buzzetto-More, 
2007). In higher education, the movement to develop accreditation mechanisms for 
higher education institutions coincided with and drove the first models of measuring 
educational outcomes (Buzzetto-More, 2007). Even today, electronic portfolios are used 
for accreditation purposes, from the unit to institutional levels (Meyer & Latham, 2008). 
Here, Buzzetto-More (2007) suggests that Northeast Missouri State University and 
Alvemo College were the first to develop assessment models to meet the evaluation 
needs of outcome-based instruction. Today, assessment is considered as an integral piece 
of higher education processes and a mechanism that ensures that students achieve their 
learning goals (Buzzetto-More, 2007). The goals of assessment in present day higher 
education include: (1) improve student learning; (2) help students to identify their 
strengths and weaknesses; (3) review and improve the quality of instructional and 
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teaching strategies; (4) improve teaching effectiveness; (5) review and improve curricula 
activities and programs; (6) provide administrative data to support decision making; and 
(7) develop effective communication channels with stakeholders (Buzzetto-More, 2007). 
Since portfolios are an important type of assessment, all these goals and procedures also 
apply to them. 

According to J.O. Brown (2001), portfolios were first used in higher education 25 
years ago and, since then, tens of thousands of adult learners have developed their own 
learning portfolios to get college credits and successfully complete their degrees. The 
term “portfolio” originates from the graphic arts, where portfolios were used to collect 
evidence to confirm that the act of learning has taken place (Snadden & Thomas, 1998). 
Today, portfolios are being extensively used in almost all disciplines, to document the 
learning progress (Snadden & Thomas, 1998). Portfolios may include records of 
experiences and events, critical reviews of articles, descriptions of teaching sessions 
attended by adult learners, projects in which adult learners participated, etc. (Snadden & 
Thomas, 1998). Portfolios enable adult learners to identify and reflect on the weakest and 
strongest sides of their learning progress, describe what they have learned and what they 
still need to leam, as well as possible ways to deal with new learning (Snadden & 
Thomas, 1998). However, Snadden and Thomas (1998) note that portfolio assessments 
work best when they are implemented and operate through continuous interactions 
between learners and instructors. Interactivity is the main prerequisite for turning 
portfolio assessments into a driver of continuous learning (Snadden & Thomas, 1998). 

The history of portfolio use in American education is integrally linked to the 
name of John Dewey, who is often considered to be one of the first proponents of 
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reflective thinking in education (Snadden & Thomas, 1998). Dewey’s contribution to 
popularization of portfolio assessments can hardly be overstated. Portfolio assessments 
were developed specifically to enhance, motivate and encourage reflection in education 
(Snadden & Thomas, 1998). Today, portfolios can be defined as “a purposeful collection 
of student work that exhibits the student’s efforts, progress and achievements in one or 
more areas” (Snadden & Thomas, 1998, p.193). However, like everything else in 
education, portfolios are susceptible to the effects of technological advancement. The 
tenn “electronic portfolio” is gradually replacing traditional forms of portfolio 
assessments. Batson (2002) writes that the term “ePortfolio” has already become 
commonplace. ePortfolio is essentially the same as traditional portfolio assessments, but 
it makes the entire process of learning much more convenient for learners and, 
simultaneously, meets the changeable demands of the technological reality (Batson, 
2002). Based on what Batson (2002) writes, ePortfolios expand learners’ reflective 
capabilities, through the inclusion of graphic content, audio and video instruments, and 
even animation. This is probably why eportfolios are becoming so popular in higher 
education. 

Common use of portfolios. A wealth of research has been completed to identify 
the best and most appropriate ways of using portfolio assessments. In the context of the 
present study, portfolio assessments in higher education present the issue of the major 
concern. Klenowski et al. (2006) write that, as of today, portfolio assessments can be 
found in virtually all phases of professional education and development. In higher 
education, portfolios are often used for the purpose of summative assessment (Klenowski 
et al., 2006). However, the body of research regarding the use of portfolios in fonnative 
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assessment is increasingly scarce (Klenowski et al., 2006). Nevertheless, in higher 
education, as well as in other learning environments; portfolio development and use is 
underpinned by the constructivist and dialogic theoretical constructs (Klenowski et ah, 
2006). The emphasis on knowledge construction in all fields of adult development and 
learning emphasize the value of portfolios as a form of summative assessment. 

Portfolio assessments are being extensively used in health and medical education. 
Grant et al. (2007) explored students’ perceptions of event analysis and portfolio 
assessment in medical learning. Driessen, Tartwijk, Vermunt and Vleuten (2004) 
analyzed the use of portfolios in undergraduate medical training and confirmed that 
portfolio assessments could become a unique and extremely useful supplement to 
traditional assessment models. Plaza et al. (2007) analyzed how reflective portfolios 
could be used in health sciences education and concluded that, despite the considerable 
advantages offered by portfolio assessments, the fundamental issues of their validity and 
reliability remained. By contrast, Kuper, Reeves, Albert and Hodges (2007) assert that 
reflective portfolios fit perfectly well in the purpose and nature of medical education, as 
there is growing understanding that medical education is better delivered through a set of 
social constructs. Medical education, as well as education in general, is no longer an 
individual process but, rather, a product of continuous interactions between two or more 
individuals and groups (Kuper et al., 2007). In social sciences, ideas about how people 
act are always culture- and context-specific, and only portfolio assessments can give 
professional educators a chance to understand the nature and implications of these 
behaviors (Kuper et al., 2007). Additionally, portfolios have been used and analyzed in 
postgraduate education (Tochel et al., 2009), as a tool of reflective learning in initial 
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teacher education (Chetcuti, 2007), as well as the instrument of literacy assessment 
(Walsh, 2009) and a useful tool of assessing second language learning (Cummins & 
Davesne, 2009). 

In today’s learning environments, the use of web-based portfolio assessments is 
becoming a popular topic of professional discussion. As education, including higher 
education, is moving towards online spaces, professional educators are becoming 
preoccupied with the task of developing web-based assessment fonns. Chang and Tseng 
(2009) analyzed the effects of web-based portfolio assessments on students’ performance 
in junior high school and found that online portfolios improved students’ reflection, goal 
setting and problem solving capabilities. Barbera (2009) further supported those results, 
by stating that e-portfolios and netfolios offered students a better understanding of the 
learning process, learning objectives, through self-revision and participation in peer 
assessment. 

In the qualitative analysis of electronic portfolios by Fitch, Peet, Reed and 
Tolman (2008), students reported that portfolios helped them to organize their thoughts, 
tie initially separated elements of their learning experiences into a coherent body of 
learning, make sense of other classes, although the electronic nature of portfolios made it 
difficult for many students to work with them. It seems that portfolios can be universally 
applied across a variety of learning contexts, although certain problems and limitations 
should not be disregarded. For example, Lucas (2007) writes that portfolio assessments 
represent an interesting self-evaluation scheme that enables students with various 
linguistic problems to address the existing deficiencies, through self-autonomy and 
learned independence, but only when portfolio assessment instruments are properly 


46 



developed and appropriately applied. According to Lucas (2007), the process of 
developing portfolios consists of four stages: collection, selection, reflection and 
projection. However, whatever the nature and application of the assessment instrument, it 
is always rooted in the constructivist learning framework and reflects the growing 
urgency of comprehension constructs in higher education. 

Portfolio assessments and self-efficacy. Portfolio assessments are rooted in the 
constructivist philosophy (Klenowski et al., 2006). Portfolios as an instrument of 
assessing learning reflect “a shift from a stress on individual responsibility for learning to 
a more collaborative view, allowing learners to identify issues in their organization and 
society which affect their learning and well-being” (Klenowski et al., 2006, p.269). It is 
through the prism of the constructivist philosophy that the link between portfolio 
assessment and students’ self-efficacy can be better understood. Researchers offer an 
insight into the effects of portfolios on self-efficacy in learning. 

Understanding the relationship between self-efficacy and portfolio assessments is 
important, since higher learner self-efficacy beliefs are fundamental to the professional 
maturation of students (Jones, 2009). Learner self-efficacy further predetennines higher 
motivation and better learning outcome expectations (Jones, 2009). Self-efficacy in 
learners is closely related to personal responsibility, effective goal setting and 
transformations: Jones (2009) suggests that self-efficient learners are transfonnative 
learners, who assume personal control over their learning progress. Through the prism of 
Bandura and Mezirow’s theories, self-efficacy drives the rapid transformation of the 
learning process, turning learners into both producers and products of their social 
environment (Jones, 2009). 
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Students construct their perceptions of self-efficacy through the nature and form 
of assessments used to evaluate their learning progress (Brown & Hirschfield, 2008). 
Students perceive assessment in four different ways: they may feel that assessments make 
them accountable; they may think that assessment is unfair and, therefore, irrelevant; they 
may perceive assessment as a good way to improve learning; or they may experience the 
sense of joy about particular forms of assessment. According to Strivens (2007), students 
feel that e-portfolio systems support fonnal learning, overall development, summative 
assessment, and transition to a different learning environment. In other words, portfolio 
assessments drive student’ self-efficacy, by creating a reflective dialogue between 
students and instructors and letting students express their achievements and concerns 
openly and constructively (Hayatdavoudi & Ansari, 2011). 

Portfolio assessments have the potential to drive students’ self-efficacy, since they 
create an atmosphere of learner-driven education; the latter is the foundational ingredient 
of the constructivist philosophy (Sajadi & Khan, 2011). Portfolios are fully compatible 
with the principles of active learning, which represents learners as those who have 
capability to realize mental processing through creativity and exploration (Sajadi & 

Khan, 2011). In this sense, the benefits of using portfolios to assess learners’ knowledge 
cannot be ignored. Portfolios have proved to be extremely effective in the construction 
and analysis of learning in ADHD students (Sajadi & Khan, 2011). Specific to ADHD 
students Mayer (2001) stated, “humans can process information into different channels of 
auditory/verbal and pictorial/visual” (p. 659 as cited in Sajadi & Khan, 2011). In other 
words, by creating a portfolio, ADHD students or participants of this study now have a 
visual aid of their work in progress. Portfolios raise students’ self-efficacy by reducing 
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their anxiety in writing (Ozturk & Cecen, 2007). As an instrument and medium of 
storytelling, portfolios play an important role in the development of learners’ literacy 
(Wan & Tanimoto, 2008). Students using portfolios to assess their knowledge experience 
greater satisfaction, being active participants of the learning process and able to reflect 
upon their progress (Wang & Liao, 2008). In these constructivist learning environments, 
portfolios help learners to manage learning and foster the evolution of lifelong learning 
priorities and skills (Little, 2009). 

Loyens and Gijbels (2008) claim that there is still a huge gap between the 
constructivist philosophy and educational practices. The fact that constructivism 
manifests in a number of ways and has more than one theoretical position makes it 
difficult to narrow the gap between educational practice and theory (Loyens & Gijbels, 
2008). Despite these difficulties, constructivism remains the dominant educational 
philosophy, and portfolio assessments look like a perfect element fitting in the 
constructivism atmosphere of knowledge delivery in higher education. 

Portfolios and comprehension. As previously mentioned, comprehension 
remains one of the most problematic elements of portfolio analysis in research, due to the 
fact that its meaning is often interpreted in quantitative terms. However, given the topic 
of this study, the relationship between portfolios and comprehension has to be better 
understood. Recent studies highlight the crucial role of portfolio assessments in driving 
students’ comprehension in different learning settings. As always, comprehension is 
understood in tenns of Bloom’s taxonomy and implies the process of translating and 
interpreting new knowledge, through project participation and presentations, 
demonstrations and explanations, as well as criticism (Todorova & Mills, 2007). Here, 
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the body of literature covering the relation of portfolio assessments to comprehension can 
be roughly divided into two categories: first, whether or not portfolios improve 
comprehension and, second, how exactly portfolio assessments can drive better 
comprehension of the learning material. 

Here, Conrad (2008) recognizes the potential of portfolio assessments to enhance 
the quality of learning and students’ progress. Taking Dewey’s educational philosophy as 
the basis, portfolio assessments promote real-world learning and help adult learners in 
distance education to master the new knowledge through reflective learning (Conrad, 
2008). From the viewpoint of Mezirow’s theory, portfolio assessments emphasize 
educators’ role in learning as helping students to engage in reflective thinking and 
helping them to redefine their understandings and insights (Conrad, 2008). Portfolio 
assessments are also recognized as a unique instrument of building collaborative ties and 
building schoolwide comprehension instruction (Au, Raphael & Mooney, 2008). 

Kariman and Moafi (2011) emphasize the positive effects of portfolio assessments on 
comprehension in prenatal training for midwives. Ha (2010) discovered that portfolio 
assessment improved learners’ self-expression capability in English learning. An 
important question is how exactly portfolio assessments improve comprehension. 

Several ways are possible. Fox, White, Kidd and Ritchie (2008) write that 
examining portfolio contents brings both students and educators towards better 
understanding of students’ learning practices and decisions. The results of portfolio 
assessments can further inform instructional and curriculum decisions in higher education 
(Fox et ah, 2008). Brown and Hirschfield (2007) further suggest that portfolio 
assessments keep students accountable for their own learning progress, loading students 
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positively to seek achievement. Based on the self-regulation theory, students who assume 
responsibility for their learning have greater chances to attain to the desired learning 
objectives (Brown & Hirschfield, 2007). Leu et al. (2009) contend that the Internet 
expands learning frontiers and defines new boundaries of learning and literacy in the 
twenty-first century; this being said, it is possible to assume that e-portfolios will become 
the defining feature of learning assessments in the digital age. 

Yet, the link between portfolio assessments and comprehension is not without 
controversy. Lafontaine and Monseur (2009) suggest the presence of a serious gender gap 
for open-ended questions used to measure comprehension through portfolio assessments. 
Another question is whether the responsibility and self-regulatory learning through 
portfolio assessments benefits learners. Settlage, Southerland, Smith and Ceglie (2009) 
make an interesting suggestion that not confidence but self-doubts are the basic driver of 
self-efficacy and comprehension in learning. Settlage et al. (2009) further claim that the 
importance of self-doubts in learning comprehension confirms the soundness of Dewey’s 
assumptions that uncertainty is the fundamental element of knowing and learning. 

In summary, portfolio assessments have a long history. Portfolios are becoming 
increasingly popular in learning, including higher education. Originating from graphic 
arts courses, portfolios slowly transcend to cover other learning settings and 
environments. Portfolios are used in medical education, health and language learning. If 
properly developed and implemented, portfolio assessments have the potential to raise 
self-efficacy and improve comprehension in all groups of learning, through reflective 
thinking, responsibility and self-control. 
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Self-Efficacy and Portfolios 


Many educators are struggling with the problem of defining the most suitable 
theoretical framework, to promote learning and enhance students’ progress toward the 
desired learning objective. Theories of Dewey, Mezirow, Piaget, Bandura and Vygotsky 
have been extensively used to motivate new, alternative ways of thinking in adult 
students. Researchers have made considerable contributions to understanding the concept 
of self-efficacy and its relation to the use of portfolio assessments. The most important 
findings in the field of self-efficacy and portfolio assessments have been described in the 
earlier section of this review. However, there is no practice without theory, and there is 
no theory without practice. The use of portfolios as an instrument of assessment and its 
implications for self-efficacy and comprehension requires a more profound theoretical 
analysis, and the learning theories of Bandura, Dewey, Mezirow, Piaget and Vygotsky 
can help to explain the mechanism of portfolio implementation and its effects on 
students’ conceptions of learning. Although learning theories of Bandura, Dewey, 
Mezirow, Piaget and Vygotsky are extremely popular in education research, their use in 
the analysis of self-efficacy-portfolio relationships is rather scarce. However, all these 
theories have the potential to inform curriculum and instructional design decisions 
regarding the use of portfolio assessments in distance education. 

Albert Bandura is rightly considered as the main source of knowledge about self- 
efficacy in education and learning. Bandura views individuals as independent, self- 
organizing, self-reflecting and proactive (Pajares, 2007), and learning is considered as a 
complex product of individuals’ interactions with the environment. In this sense, and 
based on Bandura’s theory, self-efficacy has nothing to do with learners’ actual 
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capabilities but refers to what learners think about their capabilities and achievements 
(Milstein, 2005). This is exactly what portfolio assessments are intended to achieve: to 
expose what learners think of the strongest and weakest sides of their learning progress. It 
should be remembered, that self-efficacy is contextual by nature and cannot be universal; 
self-efficacious learners may feel confident about solving their learning problems in one 
situation and unconfident about solving the same problem in an entirely different context 
(Wahab, 2007). 

Theories of Dewey, Piaget, Mezirow and Vygotsky continue this line of 
discussion. Yukawa (2005) used Mezirow’s theory of transformative learning to explore 
the effects of online narrative analysis on learners’ self-efficacy, which means that 
portfolio assessments hold the promise to transform learners’ experiences, making 
learning more inclusive, open and discriminating. Of great importance is the use of 
Dewey’s theory in the analysis of portfolios and self-efficacy, since Dewey was among 
the first to emphasize that reflection was the essential part of the learning process 
(Neumann & Oberhuemer 2007). This is actually why Neumann and Oberhuemer (2007) 
used Dewey’s theory as the basis for analyzing learner empowerment through e- 
portfolios. According to Neumann and Oberheumer (2007), portfolios place students in 
the role of being their learning and education architects, which also leads them to gain 
greater self-efficacy in learning. 

Again, and based on Piaget, the relationship between portfolios and self-efficacy 
can be explained in terms of the internal world of learners and self-regulation (Oon-Seng, 
2006). These are the aspects of Piaget’s theory that have been successfully used to 
explain the complex process of intelligent development and problem-based learning 
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(Oon-Seng, 2006). According to Vygotsky, learning begins in the social context and 
further transforms into the psychological processes driving the process of learning, which 
also means that portfolio assessments can become the major external factor of influence 
driving the development of inner self-efficacy perceptions in learners (Oon-Seng, 2006). 
Unfortunately, most of what has been said about these theories is merely assumptions that 
require further analysis and validation. However, that portfolio assessments are closely 
aligned with the ideas of constructivism cannot be denied. 

Self-efficacy and comprehension. Whether or not self-efficacy can drive better 
comprehension of the new material is an important question. The relationship between 
self-efficacy and comprehension was explored by Schunk and Zimmerman (2007), 
Caprara et al. (2008), and Wadsworth, Husman, Duggan and Pennigton (2007). In all 
these studies, Bandura’s concepts of self-efficacy formed the basis for interpreting the 
findings. In Schunk and Zimmerman (2007), a model of self-efficacy was developed to 
improve students’ reading and writing performances; as a result, better self-efficacy 
proved to greatly affect the quality of students’ achievement, further supporting the 
validity of Bandura’s social cognitive theory. Caprara et al. (2008) analyzed perceived 
self-efficacy and its potential contribution to academic achievement in school education 
and confirmed that low declines in self-regulatory efficacy were associated with higher 
grades and better retention in junior high school. Moreover, perceived self-regulatory 
efficacy in junior high school created the groundwork for improved achievements later in 
high school education (Caprara et al., 2008). Extremely important were the findings of 
Wadsworth et al. (2007) who, through the lens of Bandura’s theory, explored the relation 
of self-efficacy to comprehension in an online mathematics course. Again, self-efficacy 
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was confirmed to enhance learning achievements in online environments (Wadsworth et 
ah, 2007). 

Comprehension in the constructivist perspective. Earlier in this review, the 
problem of understanding the concept of comprehension has been raised. Although 
constructivism does not always help to solve this problem, it creates a good basis for 
reconsidering the relationship between comprehension and constructivism and its 
implications for practicing educators. Constructivism emphasizes the importance of 
experiences in creating meaning (Zarei & Esfandiari, 2008). Therefore, constructivist 
transfers of knowledge occur by engaging students in authentic tasks that are provided 
and executed in meaningful situations (Zarei & Esfandiari, 2008). Therefore, 
comprehension in the constructivist perspective can be interpreted as the process by 
which learners use theoretical tools in real world situations (Zarei & Esfandiari, 2008). 
Only application can ensure that the process of learning is effective and appropriate 
(Zarei & Esfandiari, 2008). Yuan (2007) writes that, in constructivism, learning is 
essentially about constructing and comprehending. Since knowledge is merely an 
explanation of a problem, it will progress and change with the progress of learners (Yuan, 
2007). Better comprehension of the learning material can be achieved, by aligning 
students’ objectives with those of learners, developing student-centered approaches to 
learning, and utilizing cooperation, situation, and communication to enable students to 
realize their potential and creativity (Yuan, 2007). The emphasis on cooperation in 
learning and its importance for comprehension is also made by Ziyaeemehr (2012), who 
refers to Vygotsky and recommends that collaborative activities can enhance students’ 
comprehension and, simultaneously, activate comprehension strategies proposed by 


55 



students. Unfortunately, only one study has explored the meaning of comprehension in an 
online graphic design course: Maldonado, Lee, Klemmer and Pea (2007) analyzed 
patterns of collaboration and their implications for comprehension in design courses. 
Again, collaboration appeared to be a serious factor of comprehension improvements in 
the online course (Maldonado et ah, 2008). The lack of research into comprehension in 
graphic courses justifies the relevance and feasibility of this study. 

Summary 

The current state of literature provides a wealth of information regarding the 
relevance and use of portfolios in education. It would be fair to say that the history of 
portfolio assessments is inseparable from the history of assessment in general. Since the 
goal of assessment in higher education is to identify students’ strengths and weaknesses, 
review and validate the effectiveness of various teaching strategies and inform 
administrative and instructional decision making (Buzzetto-More, 2007), portfolio 
assessments have all chances to become a viable mechanism of assessing learners’ 
progress in higher education. 

In light of everything said in this review of literature, several important 
conclusions can be made. First, the history of portfolios dates back into several decades, 
and their initial goal and purpose is to encourage and empower students to evaluate their 
knowledge and learning progress through reflection and critical thinking. Today, 
traditional portfolios are giving place to new, technological forms of assessing students’ 
knowledge. Unfortunately, the use of portfolio assessments in distance education is still 
poorly understood. The current state of literature suggests that portfolios as a tool of 
assessment have become extremely popular in almost all disciplines, from medicine and 


56 



health education to language and literacy development. Unfortunately, and despite the 
fact that portfolio assessments have their roots in graphic design courses, contemporary 
researchers do not pay much attention to the potentiality of portfolio assessments in 
graphic design courses, especially in online environments. Most probably, the use of 
portfolios in graphic design education is so widespread that researchers do not consider it 
interesting and useful. However, as the entire system of education moves away from 
process- towards outcome-oriented assessment philosophies and technological 
advancement changes the nature of knowledge delivery, the need to reevaluate the 
applicability of portfolios in distance education is getting more urgent. This gap in 
empirical literature further justifies the importance of analyzing the usefulness and 
usability of portfolio assessments in online learning environments. 

Second, self-efficacy and comprehension have been extensively explored in the 
context of constructivist learning. The current state of literature suggests that 
constructivism can readily become the dominant framework used by professional 
educators to design and implement learning solutions. Based on everything written and 
said about self-efficacy and comprehension, it is clear that Bandura’s social cognitive 
theory is the most popular and, simultaneously, the most relevant instrument of 
interpreting self-efficacy research findings. Then come the constructivist and learning 
theories of Dewey, Piaget, Vygotsky and Mezirow. Constructivism, portfolios and self- 
efficacy create a complex triangle of interdependencies: while portfolios have the 
potential to enhance learners’ self-efficacy through openness, reflection, and storytelling, 
constructivism further justifies the use of portfolio assessments in learning, with the goal 
of making students responsible for their learning progress and, consequentially, making 
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learning meaningful and productive. The positive relationship between self-efficacy and 
comprehension has been abundantly established. While the Internet and new 
communication technologies expand higher education frontiers, the meaning of 
comprehension and its relation to self-efficacy and portfolio use in distance education 
should be reviewed. 

Third, theories of constructivism lay the groundwork for re-interpreting the 
relationship between self-efficacy and comprehension and have the potential to improve 
professional understanding of portfolio use and its implications for learners in distance 
learning. In constructivism, as well as in the rest of recent studies, comprehension is 
usually understood as part of Bloom’s learning taxonomy and the process by which 
individuals recognize, analyze and apply new knowledge in real-world environments. 
Applicability stands out as the central criterion of comprehension in classroom-based and 
distance education settings. Comprehension emphasizes the learner-centered nature of 
knowledge development and learning and implies that portfolio assessments could help 
students to systematize and organize their knowledge, making learning meaningful and 
important. 

Finally, a large body of literature is devoted to the analysis and evaluation of 
qualitative methodologies in the study of portfolio assessments, self-efficacy and 
comprehension. Contemporary researchers are almost unanimous in that qualitative 
frameworks best fit in the needs and requirements of portfolio research. Reasons why 
qualitative analysis reflects and suits the nature of portfolio assessments are numerous 
and diverse. Basically, qualitative analysis reinforces the storytelling and narrative nature 
of portfolio assessments, as well as the crucial role of reflection and critical thinking in 
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their development. Qualitative research is the best way to uncover and explore the hidden 
messages sent by distance learners through portfolios. Quantitative instruments cannot 
capture the complexity and intricacy of self-reflection in portfolio assessments. Likewise, 
qualitative methods of research are the best solution to methodological problems inherent 
in the study of self-efficacy: because self-efficacy is essentially about how students 
perceive their own competencies and capabilities, qualitative methods can expose 
complex changes in students’ perceptions of their progress under the influence of 
portfolio use in distance education. 
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CHAPTER 3. METHODOLOGY 


Introduction 

The field of higher education has been in the state of transition from brick and 
mortar to technological outcome-oriented processes and decisions (Zawacki-Richter, 
Hanft & Backer, 2011). The lack of attention to competency-based assessments and the 
peculiarity of distance education, as well as the growing utilization of portfolios in brick 
and mortar higher education classes justified the need to conduct this research. As such, 
the purpose of this comparative case study was to explore the impact of portfolio 
assessments on students’ perceptions of self-efficacy and comprehension in a graphic 
design course. The current state of literature confirmed that assessment tools used by on- 
ground institutions differ greatly from those used by online institutions in the higher 
education field. Benefits offered by portfolios in fonnative assessment are numerous 
(Chung, 2008; Kicken, Brand-Gruwel, Merrienboer & Slot, 2009; Smith & Tellema, 
2007; Stansberry & Kymes, 2007; Venable, 2010; Zawacki-Richter et ah, 2011). This 
study was focused on the analysis of portfolio assessment and its potential to enhance 
comprehension and students’ self-efficacy in an online design course. 

Research Questions 

Since the goal of this study was to understand whether or not portfolio 
assessments positively affected students’ self-efficacy and comprehension in an online 
design course, the following research questions will have to be answered: 

1 . How do portfolios impact students in an online design course? 
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2. How do portfolios impact students’ perceptions of self-efficacy? 

3. How do portfolios impact students’ comprehension? 

The study built on the benefits and features of qualitative research design, which 
has proven to be extremely valuable in the analysis of portfolio use in higher education 
(Stansberry & Kymes, 2007). According to Creswell (2009), qualitative research is a type 
of research where researchers rely on participants’ views. This was why qualitative 
analysis was at the heart of this study. This case study was focused on portfolio 
assessment in the online graphic design-learning environment in order to provide a 
framework for future teachers who are teaching within the arts field. 

Research Design 

Case study was the foundation of the research design for this study. Woodside 
(2010) defines case study research as “empirical inquiry that investigates a contemporary 
phenomenon within its real life context, especially when the boundaries between 
phenomenon and context are not clearly evident” (p.l). Case study research is actually a 
type of research that focuses on understanding and describing, controlling or predicting 
individuals, processes, organizations, etc. (Woodside, 2010). The main distinguishing 
feature of case study research is in being focused on individuals (Woodside, 2010). 
Speaking figuratively, a case study researcher will never study a thousand rats for one 
hour but, instead, will focus on studying one rat for a thousand hours - this is one of the 
most famous statements of B.F. Skinner (1966) cited in Woodside (2010). Certainly, that 
does not mean that case study is the study of one individual over a lengthy period of time; 
rather, the key point of case study analysis is in digging up the bones of knowledge and 
consciousness in the individuals participating in the study. Case study is the most useful 
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when a profound analysis of changes in individual perceptions, feelings and knowledge 
have to be made (Woodside, 2010). 

Case studies are extensively used in fields and contexts where the boundaries 
between the phenomenon in question and the context in which it occurs are blurred 
(Darke, Shanks & Broadbent, 1998; Yin, 1994). It should be noted, that case study is a 
method of research that focuses on the analysis of one single setting (Eisenhardt, 1989). 
Case studies are often claimed to combine several different research methods, although 
the use of one method of analysis is quite possible (Woodside, 2010). Unfortunately, the 
use of case studies in educational research is associated with several major 
misconceptions. Flyvbjerg (2006) lists five major misunderstandings about the nature and 
implications of case study research. First, case study research tends to be associated with 
theoretical, not practical knowledge and researchers who seek to choose the most 
appropriate analytical method erroneously assume that case studies place theoretical 
knowledge over practical findings (Flyvbjerg, 2006). Second, single case-studies are 
claimed to be non-generalizable (Flyvbjerg, 2006). Third, a popular assumption is that 
case studies are more appropriate for generating hypotheses, whereas other methods of 
analysis should be used to test and validate (or invalidate) these hypotheses (Flybjerg, 
2006). Fourth, case studies are often blamed for their bias toward verification (Flybjerg, 
2006). Fifth, specific case studies are difficult to summarize (Flybjerg, 2006). All these 
controversies reduce the frequency with which case studies are used in education and 
other fields of research. Yet, these are merely misunderstandings. Moreover, it would be 
fair to say that a science/ discipline without a great number of professionally perfonned 
case studies is actually a discipline that lacks systematization in the production and use of 
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exemplars (Flybjerg, 2006). Any discipline without a good deal of exemplars is actually 
an ineffective one (Flybjerg, 2006). To a large extent, this study did not simply use case 
study as the central methodological framework but also shows that applying case study 
methods in educational research is easy, possible, realistic and useful. The benefits and 
usefulness of case study research have been described in abundance and confirm the 
relevance of the case study choice in this research. 

McLeod (2010) asserted that, for anyone embarking on case study research 
methods, it is important to be aware of case study advantages. The usefulness and 
usability of case studies in research have been widely established, mainly due to the 
flexibility and critical perspectives provided by case studies (McLeod, 2010). Case study 
research makes possible the analysis and description of innovative practices and unusual 
cases (McLeod, 2010). Through case study research, successful integration of learning 
and practice becomes possible (McLeod, 2010). Eventually, the use of case study designs 
is justified by the opportunities provided by case studies in uncovering the subconscious 
processes affecting individuals in their routine practices (Woodside, 2010). Unlike 
traditional surveys and interviews, case studies are inherently intended to expand the 
researcher’s access to participants’ thinking processes (Woodside, 2010). In this sense, 
the proposed research design was particularly useful for the analysis of students’ thinking 
and self-efficacy perceptions and processes: bearing in mind the overall complexity of the 
self-efficacy concept, it was possible to assume that not all students would be able to 
explicitly define the changes in their perceptions under the influence of portfolio 
assessments in an online graphic design course. Case study design will have enabled 
effective measurement of the ongoing thinking processes in the research participants 
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(Woodside, 2010). Eventually, case studies have proved to be an effective instrument of 
research and analysis in the field of education, including distance education and virtual 
educational domains (Baxter & Jack, 2008; Girvan & Savage, 2010; Liaw, 2008; 
Reingold, Rimor & Kalay, 2008; Samarawickrema & Stacey, 2007). 

Case study method reflected the inquiry-based direction of this study and 
facilitated the analysis of complex real-life environments. With the help of the case study 
method, the rich problematic of portfolio utilization in distance education can be 
uncovered (Flyvbjerg, 2006). For students and novice experts, knowledge generally 
consists of intimate experiences drawn from thousands of individual cases that cannot be 
standardized (Flyvbjerg, 2006). By operating at the level of experience, researchers use 
real expertise to judge the differences and contradictions in case study narratives 
provided by research participants (Flyvbjerg, 2006). The use of case study design ensures 
the researcher’s proximity to the real-life environments in which distance learning 
occurs; this proximity, in turn, creates conditions that favor advanced understanding of 
case problematic and results in more discoveries than any other method of research may 
allow (Flyvbjerg, 2006). Case study was the best choice for this study, since only case 
study questions and process could provide an in-depth understanding of the issue in 
question (Creswell, Hanson, Plano & Morales, 2007). 

Of critical importance is the issue of qualitative validity and qualitative reliability 
in case studies. The main criteria for assessing the rigor of case study research include 
internal validity; construct validity; external validity; and reliability (Gibbert, Ruigrok & 
Wicki, 2009). Objectively, the case study method contains no more bias toward 
verification of its results than other methods of inquiry (Flyvbjerg, 2006). Therefore, it is 
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within any researcher’s capability to guarantee that the use of case study methods leads to 
valid and verifiable results. In this study, case study methodology was the best and most 
valid instrument of answering the research questions. Case study method is an instrument 
of inquiry - an inquiry that has already become a distinguishing feature of contemporary 
learning environments (Fallon, 2011). Moreover, while discussing their self-efficacy and 
comprehension levels, students may fall short of cognitive resources to judge their 
abilities and talents more objectively (Woodside, 2010). The emphasis of this research 
was to uncover the elements of self-efficacy and comprehension that may have been 
hidden from the eyes of the research participants. It is the type of research that takes 
place in one or more bounded environments, among multiple actors that directly or 
indirectly participate in these contexts (Woodside, 2010). Based on Simon’s (1990) claim 
that human behaviors are shaped by the scissors whose two blades include the structure 
of the environment and the cognitive capabilities of the actor (as cited in Woodside, 
2010), case study methods enable to account for both factors and bring them together to 
shape an objective picture of changes in students’ self-efficacy perceptions and 
comprehension, as a result of portfolio use in an online graphic course. In this study, 
objective comprehension and subjective self-efficacy beliefs shape the scissors that 
govern student behaviors in an online graphic course and predispose their successes and/ 
or failures in technologically guided learning. 

Setting of Study 

According to Creswell (2008), sampling for case studies can range from “1 or 2 to 
30 or 40” (p. 217). For this study, a random sampling was completed using two classes 
with each class having an enrollment between 20 and 25 students. Creswell (2008) 
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reminds us that random sampling provides an equal probability for any of the students to 
be chosen to participate as participants within this study and by doing so, “ensures that 
the sample will be representative of the population” (Keppel, 1991, as cited in Creswell, 
p. 155). The perspective sample for this study was representative of one design course, 
which averages between 20 and 25 students. 

According to Creswell (2008), this procedure would allow the researcher to 
randomly assign individuals from both classes and assign to separate groups. Cohen 
(2006) defines random sampling as “a systematic process of selecting subjects or units 
for examination and analysis that does not take contextual or local features into account” 
(para. 1). Students choosing to participate were referred to as participants, likewise, 
students choosing not to participate or students not chosen to participate were referred to 
as the control group. 

The main site for this study and research was an online university where students 
attended two separate Digital Imaging design courses within the Graphic Arts 
undergraduate degree program. The format for this study consisted of 20 participants in a 
graphics design undergraduate course entitled, Digital Imaging. The course ran for five 
weeks where students were required by the School of Education (SoE) to meet specific 
criteria prior to entering this course and in doing so ensured the same skill levels of each 
student entering this course. For this specific course, there were no prerequisites required 
to enter this course. In other words, no previous training was required. This was an 
introduction level, beginning course for their undergraduate degree program within the 
field of Graphic Design. 
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Students were randomly selected and assigned to one of the two research groups. 
A faculty member from the SoE completed the process of selecting participants. This was 
one of the best ways to avoid sampling bias. Random sampling was used to choose 
research participants, who were randomly assigned to either the study or the control 
group. The faculty member from the SoE was asked to follow a simple procedure of 
random sampling, by printing the names of potential participants and picking them out of 
a hat. Only the first 20 students were given a chance to voluntarily participate in the 
study. If, for some reason, a student chose not to participate in the study, another name 
was chosen, until 20 names were ready to be included in the list of the study respondents. 
By choosing all students from one and the same course, the researcher was able to ensure 
greater representativeness of the study sample. 

Once the SoE successfully identified the participants, the researcher was notified 
by email that the study consisted of 20 participants. The SoE sent out surveys during 
weeks one, three and five through the schools’ email system and also received the 
participant’s responses (Appendices A-B) in the same manner. Once the participants and 
the control group returned their surveys, the SoE separated these responses and provided 
a multi-page PDF that was sent to the researcher. One PDF contained the responses from 
the participants and another PDF was provided with the results of the control group. The 
survey items related to the research questions posed for this study. Having the SoE 
choose the participants and receive their responses eliminated any researcher bias within 
the study, as the researcher was also the instructor for these two courses. Once the SoE 
collected the responses during their respective weeks, the SoE sent one email with the 
participant’s responses and another email with responses from the control group. The 
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documents contained no identifying marks that allowed the researcher to identify the 
participants. 

Participants - Group A differed only because they created a portfolio during all 
five weeks of their design work. Each week during lecture, the instructor demonstrated 
the processes necessary to create a portfolio. For example, during week one, the 
participants would save their design work with the requirements stated for that week’s 
design project. Upon completion of week one, the participant saved their work as a multi- 
page Portable Document Format File (PDF). Each week thereafter, the participants 
followed the same format for weeks two through five. Adding each week’s designs to one 
document, allowed the researcher to provide a grade based on the assigned rubric 
(Appendix D). Once the instructor provided a grade for the portfolios, they were sent to a 
faculty member within the SoE who either agreed with the grade provided, or provided a 
different grade followed with comments including an additional evaluator provided 
legitimacy to this study and its data. 

Non-Participants - Group B was used as the control group. Because the instructor 
was not privy to the identity of either group within this study, Group B received the same 
information and feedback during lectures as Group A. The only difference was that the 
participants’ (Group A) created a portfolio and the non-participants (Group B) did not. 

In both classes, participants and non-participants completed their discussion board 
(part one), by day three of each week, with their final discussion due by day 7. Also due 
by day 7 was their required design project for the week. Formative feedback was 
provided to both groups. The instructor also provided two one-hour live lectures that 
demonstrated necessary activities within their weekly projects that would enable a 


68 



successful design with the requirements posted within each assignment. Live lectures 
were conducted using Adobe Connect, which was included within students’ classrooms. 
Participants and non-participants also received a weekly grade entered into their 
gradebook. The non-participants were given written feedback, which included positive 
feedback, as well as critical written formative feedback that provided recommendations 
to enhance their skills. Other tools available to both groups, were the opportunity to email 
their instructor or contact their instructor by phone should they need clarification or help 
with any aspect of their assignments. 

Recruitment took place prior to the beginning of their class in the form of email to 
each of these students. Students were infonned that their participation was voluntary and 
would not affect their grade whether they chose to participate or chose not to participate. 
They were adequately informed as to the process of the random sampling, as well as 
explained that they would not be providing their responses to their instructor. Instead, 
they would be in direct contact with the SoE in order, to protect their identity within the 
study or fear of bias, as the instructor is also the researcher. 

The combination of the proposed sampling procedures and a relatively small 
sample size is typically characteristic of all case studies, case studies usually focus on a 
small number of participants, which, in turn, complicates randomization (Gerring, 2007). 
If a sample is made of a large number of participants and their cases, the selected cases 
may not be representative of the overall population (Gerring, 2007). By contrast, when 
cases and research participants are distributed homogeneously across all study groups and 
variables; the probability that cases from all-important segments of the analysis will be 
included in the sample dramatically increases (Gerring, 2007). The proposed sampling 
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procedure and the number of research participants leveraged the amount of information 
needed to perform productive causal analysis. In other words, with random sampling, it 
provides a cause and effect relationship between participants and the control group. By 
performing a random sampling, each student has a 50/50 chance of being assigned as 
either - a participant or being assigned into the control group. Unlike other methods of 
analysis, Gerring (2007) recommends that the goals of case study research be met 
through purposeful selection of the research participants. In this study, the selection 
procedure allowed both (a) effective randomization and (b) representation of the target 
population. 

Despite the effectiveness of the discussed sampling procedures, the risks of non- 
response bias were not to be disregarded. Respondents chosen to participate in this study 
may have lacked motivation or time to provide adequate responses or may have had the 
fear that their responses might impact their grades, despite being assured by the 
researcher that their participation would not influence their learning outcomes in any way 
(Gratton & Jones, 2010). To enhance the quality of the sampling procedure, all 
participants will be asked to report their concerns and fears that may have affected their 
motivation to provide case study responses, prior to taking participation in the study. By 
addressing these concerns, the researcher was able to reduce the risks of non-response 
bias and ensure greater legitimacy of the proposed methodological framework. All 
students received follow-up letters to assure them that their agreement to participate in 
the study will not affect their grades. These letters signified the completion of the 
sampling stage of the study. 
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Instrumentation 


Identifying and negotiating access to respondents and sites was one of the crucial 
stages in the design and implementation of the research project. One of the major tasks 
faced by qualitative researchers is to increase the likelihood of choosing good sites and 
secure respondents’ participation in the project (Devers & Frankel, 2000). Since in 
qualitative research the researcher is also the research instrument, the study requires that 
the researcher develops, maintains and closes productive relationships with research 
subjects and research sites (Devers & Frankel, 2000). This study was an example of the 
research project where the choice of a good research site was obvious as it met the goals 
and fits the nature of the study. Pennission for the study was gained in a personal contact 
with the Faculty Member in a School of Education in the participating university. The 
university is a field of research that requires permission to be studied (Yin, 2010). This is 
the private space whose boundaries are clearly defined and which necessitates obtaining 
official pennission to perform the study. Yin (2010) stated, “Public schools are private in 
the sense that you will need permission from school officials to conduct research as well 
as pennission from those officials and parents if you want to converse with or take 
pictures of any of the students” (p. 1 13). That the research project takes place in the 
university setting implies that no pennission from students’ parents needed to be gained, 
but was essential that official permissions were gained from both the university officials 
and the respondents who expressed their willingness to participate in the study. 

The researcher’s qualifications were sufficient to design and implement this 
professional research project and achieved sufficient qualitative reliability and qualitative 
validity of the study findings. Qualifications play one of the central roles in achieving 
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quality results. Researchers’ qualifications matter for the institutional boards that approve 
or disapprove the research proposal; simultaneously, the presence of sufficient 
qualifications guarantees that the researcher does not damage the image of the institution 
where the study takes place and does not do any harm to the study participants (Streubert 
et ah, 2010). The researcher clearly understands the role and importance of the study and 
its potential effects on the research participants (Streubert et al., 2010). 

The risks of the researcher’s bias should not be disregarded. Bias is described by 
Stake (2010) as undesirable and ubiquitous. Becoming a qualitative researcher involves 
better understanding of how to deal with personal bias (Stake, 2010). All researchers, 
people, reports and stories have their biases, and the principal goal of quality research is 
to identify and reduce the influences of these biases on the results (Stake, 2010). The 
researcher’s learning experiences and opinions regarding students’ self-efficacy and 
comprehension in an online design course, as well as personal impressions from using 
portfolios as an instrument of assessment could have greatly affected and altered the 
picture of respondents’ answers during the study. To avoid these controversies and 
misunderstandings, the SoE sent out and received the surveys to both the control group as 
well as the participants during weeks one, three and five. This eliminated any researcher 
bias within the study. The survey addressed the following questions: 

1 . How do portfolios impact students within a design course? 

la. how do portfolios impact students’ perception of their self-efficacy? 

lb. how do portfolios impact a student’s desire to create a portfolio for other 
design courses? 
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The SoE in turn, provided those responses (free from any identification) to the 
researcher each of those three weeks, where the infonnation was entered into a database 
provided by SurveyMonkey. Once the infonnation was recorded from the database for 
the comparative case study, it was implied that the results answered the research 
questions, as well as provide any insight to new, emerging themes. The researcher 
anticipated that the results of the study would enhance the quality of portfolio 
assessments and increases their use in distance courses. The researcher also anticipated 
that the experiences gained by students in this course would guide them through the 
development of better knowledge and skills in other fields. 

For this comparative case study, no certified instruments were available for use. 
As a result, the researcher conducted a field test of an open-ended survey that 
appropriately measured the perceived self-efficacy of both the control group and the 
participants (Appendices A-B). Also field-tested was a set of rubrics that successfully 
measured comprehension levels of the skills and design concepts for the given design 
course (Appendices C-D). Field tests were completed prior to the data collection process. 

Prior to the beginning of their design course, students entering the Digital 
Imaging undergraduate graphic design course were sent a welcome email to their course. 
Within the email was an introduction to the research, outlining the importance of students 
responding to any emails from the SoE. They were notified that their participation in 
creating a portfolio for their design course was absolutely voluntary and they would have 
the option to opt out if their name was chosen. 

Prerequisites for this course included having completed and earned a high school 
diploma (or its equivalent), plus students were required to complete entry-level computer 
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courses when entering into the online learning environment. These requirements 
introduce students to the online classroom, email and instruct students how to maneuver 
within their classrooms, and how to use the basic programs such as Microsoft Office 
before moving into the graphic design courses such as Illustrator, Photoshop, InDesign 
and web design. Students entering into Digital Imaging were required by the university to 
have met these specific prerequisites for the course, placing everyone at the same levels 
of skill and comprehension at the beginning of this course. It was with the creation of the 
portfolio that the participants going through this process that demonstrated increased self- 
efficacy in design, knowledge and comprehension of their given software, and increased 
levels of perceived self-efficacy from unit one - to unit five of their course when 
compared to the control group. 

Both the control group and the participants were graded using the same rubrics for 
this course (Appendices C-D), which measured comprehension of the concepts of design, 
as well as comprehension of the skills required and learned throughout the design course. 
The instructor and researcher for this case study was teaching both courses 
simultaneously. 

The researcher in this study provided an open-ended survey that responded to the 
research questions. Prior to the start of the undergraduate Digital Imaging course, the 
researcher conducted a field test by sending the following instruments to the SoE: 

1. Open-ended survey for control group (Appendix A); 

2. Open-ended survey for participants (Appendix B); 

3. Rubrics that measured both comprehension of concepts and skills of Digital 
Imaging (Appendices C-D). 
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The researcher conducted a field test by sending the required instruments for this 
study to the SoE of the university. Both faculty and staff assessed the strength of the 
rubrics and the questionnaires to ensure its reliability for this research and evaluated these 
instruments. The need for an independent assessment of rubrics was justified by the 
controversy surrounding the validity and reliability of rubrics as a fonn of learning 
assessment (Rezaei & Lovorn, 2010; Jonsson & Svingby, 2007). Within the field test, the 
instructor/researcher requested an assignment from another instructor who has previously 
taught the same course. The assignment was graded per the rubrics provided for 
measuring the comprehension of design concepts and comprehension of skills of design 
(Appendices C-D). When completed, the researcher forwarded the assignment to the SoE 
for them to also assess the assignment according to the rubrics. Likewise, the same 
experts evaluated and assessed the strengths and weaknesses of the open-ended surveys 
designed to provide data for the comparative case study (Appendices A-B). Jarobe (2009) 
stated, “Field tests are often recommended to assess the strength of research 
questions/hypothesis or test the appropriateness of certain data-gathering protocols (e.g., 
informed consent procedures), instruments/tools (e.g., item content) and data analysis 
procedures” (para. 2). Once experts reviewed the instruments, the researcher moved 
forward with the International Review Board (IRB) process. 

Data Collection Procedures 

The following instruments were used to collect the data (refer to Appendices A- 
D): participant self-efficacy survey, control group self-efficacy survey, and rubric to 
measure comprehension of design skills and design concepts. All instruments of data 
collection were field-tested prior to their implementation in the research project. The data 
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collected during the study was provided through participant surveys. The data was to be 
collected from the 20 students randomly chosen by the SoE, who had provided their 
agreement to voluntarily participate in the study. As previously mentioned, the researcher 
would serve as the central instrument of data collection, and the human-as-instrument 
philosophy would lay the groundwork for the implementation of this study project 
(Klenke, 2011). The phenomena to be observed include comprehension and self-efficacy, 
and the scope of these phenomena to be measured during the study are defined in the 
surveys distributed among the participants. 

A faculty member from the SoE from the participating university will select the 
students to be used within the study. Participants were chosen from each class on a 
random basis. Student’s names were printed and cut up and placed into a hat. The first 20 
names selected were approached to participate within this study on a volunteer basis. If a 
student opted out of the study, another name was chosen to help fill the gap. If a student 
opted out of participating within this study, that student became part of the control group. 
Ensuring the 20 participants represented one design course and therefore be a successful 
representation of that design course. 

Once the SoE had successfully identified the participants, the researcher was 
notified by email that the study consisted of 20 participants. The SoE will sent out 
surveys during weeks one, three and five and also receive the participant’s responses 
(Appendices A-B). The surveys responded to the research questions posed for this study. 
By the SoE choosing the participants and receiving their responses to the survey 
eliminated any researcher bias within the study, as the researcher was also the instructor 
for these two courses. Once the SoE had the responses during their respective weeks, the 
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SoE sent one email with the participant’s responses and another email with responses 
from the control group. The documents contained no identifying marks that allowed the 
researcher to identify the participants. 

Participants - Group A differed only because they created a portfolio during all 
five weeks of their design work. Each week during lecture, the instructor demonstrated 
the processes necessary to create a portfolio each week. For example, during week one, 
the participants saved their design work with the requirements stated for that week’s 
design project. Upon completion of week one, the participant saved their work as a multi- 
page Portable Document Format File (PDF). Each week thereafter, the participants 
followed the same format for weeks two through five. By adding each week’s designs to 
one document, allowed the researcher to provide a grade based on the assigned rubric 
(Appendix D). Once the instructor had provided a grade for the portfolios, they were sent 
to a faculty member within the SoE who either agreed with the grade provided, or 
provided a different grade followed with comments including an additional evaluator, 
provided legitimacy to this study and its data. 

Non-Participants - Group B was used as the control group. Because the instructor 
was not be privy to the identity of either group within this study, Group B received the 
same information and feedback during lectures as the Group A. The only difference was 
that the participants’ (Group A) created a portfolio and the non-participants (Group B) 
did not. 

In both classes, participants and non-participants completed their discussion board 
(part one) by day three of each week, with their final discussion due by day 7. Also due 
by day 7 were their required design project for the week. Feedback was provided to both 
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groups. The instructor provided two, one-hour live lectures via Adobe Connect, which 
was included within students’ classrooms. Participants and non-participants also received 
a weekly grade entered into their gradebook each week. The non-participants were 
provided written feedback, which also included positive feedback, as well as critical 
written, formative feedback that provided recommendations to enhance their designs. 
Other tools available to both groups, was the opportunity to email their instructor or 
contact their instructor by phone should they need clarification or help with any aspect of 
their assignments. 

Recruitment took place prior to the beginning of their class in the form of email to 
each of these students. Students were infonned that their participation is voluntary and 
would not affect their grade whether they chose to participate or chose not to participate. 
They were adequately informed as to the process of the random sampling process, as well 
as explaining they would not be providing their responses to their instructor, but directly 
to the SoE to protect their identity within the study for fear of bias as the instructor was 
also the researcher. During each of the five weeks, participants will create a portfolio of 
their required work created on a weekly basis. They were also given an open-ended 
survey during weeks one, three and five of their course (reflecting the participant’s 
perception of their self-efficacy, as well as demonstrating their levels of comprehension 
of their portfolio creation process). In week five of the participant’s course, they 
completed their final analysis of the portfolio creation process. It is through the data 
collection phase that the measurement of their self-efficacy was evaluated and measured. 

Both participants and control group received two, one-hour lectures provided by 
the instructor, who was also the researcher, which demonstrated varying processes 
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students needed to complete their weekly design task. This fonnative assessment was 
meant to provide personalized insight as to how each student might improve upon their 
designs. They were also provided written feedback to each of their discussion board on a 
weekly basis. Both groups received instruction and formative feedback in the same 
manner. 

Comprehension levels of the design course was measured for both groups by 
implementing rubrics that will measure comprehension of required design skills, as well 
as comprehension of the concepts of design. 

Before the surveys were distributed among the participants, pre-assessments of 
self-efficacy and comprehension were perfonned. The goal of pre-assessments was to 
measure the levels of self-efficacy and comprehension and use these data as covariates in 
the analysis of covariate procedure. The results of these pre-assessments enabled the 
researcher to achieve greater validity and reliability of the study results and, 
simultaneously, raise the comparability of the data obtained during the study. Despite the 
fact that Bryman and Hardy (2009) claim that the comparability of data can never be 
assumed, pre-assessments would ensure that the results could be effectively compared 
and used as an instrument of future theory-building. It is important to remember that 
formative assessments were provided to measure students’ progress during the process of 
learning. What strategies would be effective for formative assessment in distance 
education was beyond the scope of this analysis. The choice of the exact fonnative 
procedures was to be made by the SoE. Nevertheless, fonnative assessment was to 
become one of the foundational pillars in this research project (Wang, 2007). The results 
of formative assessments enabled the researcher to identify possible gaps and 
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inconsistencies in the development of self-efficacy and comprehension in the research 
participants. 

Data Analysis Procedures 

Creswell (2008) stated that an effective protocol would be to create a matrix or a 
“table of sources” to effectively organize your material (p. 245). Stoddart (2006) 
provided two similar surveys. One survey was created for the control group and the 
second survey - created for the participants created the portfolios. SurveyMonkey - an 
online software program that has a built-in analysis tool within its program provided data 
analysis. The researcher also used HyperREARCH, a qualitative data analysis program. 
With this software, the researcher was able to “code, retrieve, build theories, and conduct 
analysis of qualitative data” (Stoddart, 2006, p. 39). By creating the multiple methods to 
analyze the data provided the validity of the study (McMilan and Schumacher, 2001 as 
cited in Stoddart, 2006). 

Upon completion of the five weeks, the researcher generated a few broad themes 
or categories and provided “evidence for each category” (p. 248). The final analysis was 
completed by providing a case study focusing on the perceived self-efficacy of the 
participants; and a second narrative regarding the participant’s levels of comprehension 
of design and the process of portfolio creation throughout the course. 

For this comparative case study, three instruments were created by the researcher; 
1) Open-ended survey for the control group (Appendix A); 2) Open-ended survey for the 
participants (Appendix B), and 3) Rubrics measured comprehension of design skills and 
comprehension of design concepts (Appendices C-D). A field test was conducted by 
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sending the prescribed instruments to the SoE in order for them to assess the strengths 
and weaknesses of the prescribed instruments. 

A combination of multiple methods of data analysis would “ensure the validity of 
the study” (McMillan & Schumacher, 2001 as cited in Stoddart, 2006, p. 39), such as the 
use of SurveyMonkey - that would analyze the data entered into its database by the 
researcher. Graphs were generated based on participants’ responses. The SoE in turn, 
provided those responses from both groups (free from any identification) to the 
researcher each of those three weeks, where the infonnation was entered into a database 
provided by SurveyMonkey. Once the infonnation had been recorded from the database 
for the comparative case study, it was insinuated that the results would answer the 
research questions, as well as provide any insight to new, emerging themes 

From SurveyMonkey, responses from the both groups were then entered into 
HyperResearch software, which is a qualitative data analysis program that allows the 
researcher to “code, retrieve, build theories and conduct analyses of qualitative data” 
(Stoddart, p. 39). From the narrative responses provided by the participants and control 
group, common words were chosen as keywords and placed into HyperResearch where 
this was then coded, producing common themes from the surveys and provided new 
themes for future research. Baker (2003) talked about questionnaires as being, “the 
principle means used for collecting data by means of a survey of a designated population 
or sample in which the researcher is interested” (p. 343). 

The participant’s portfolio analysis was completed, beginning with the completion 
of the participants’ five- week course. The analysis was in the form of a comparative case 
study that provided insight from the participants (taken from the surveys each week) as to 
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how they interpreted the process of creating a portfolio for their Digital Imaging course. 
Shulman (2003) discussed five benefits in the use of portfolios as being; 1) portfolios 
pennit tracking of longer periods of teaching verses a single observation; 2) portfolios 
can bridge gaps between the process and the product; 3) “portfolios institutionalize norms 
of collaboration, reflection and discussion”; 4) a portfolio provides structure; and 5) a 
portfolio shifts the responsibility of teaching - to learning - back to the participant 
(constructivist learning) (as cited in Lombardi, 2008, p. 9). Comparisons and analysis was 
used to determine both groups’ self-efficacy and comprehension levels throughout their 
course. 

Ethical Considerations 

The main consideration for this study and for the participants was the 
confidentiality of selected participants and control group. Confidentiality was one of the 
most controversial aspects of practical qualitative research. Gratton and Jones (2010) 
suggested that all participants should be informed as for who, when, where and how will 
access and use the infonnation provided by the research participants (Gratton & Jones, 
2010). Qualitative researchers must ensure that, once the data are collected, no one will 
have access unless they were authorized to do so (Gratton & Jones, 2010). To ensure 
confidentiality of the research, only the researcher and the SoE participating in the study 
had access to the primary data. All research participants were provided with a written 
guarantee to maintain the confidentiality of their data. No personal information was 
collected. Maintaining confidentiality was protected with all forms of communication 
including emails, which was sent by the SoE during weeks one, three and five. The SoE 
also received responses from both groups in response to the given surveys. 
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Informed consent and gaining infonned consent from the research participants 
was at the heart of the ethical nature of this study. Providing the research participants 
with the fullest infonnation about the benefits and foreseeable risks of the study was 
central to the operationalization of the infonned consent phenomenon (Murphy & 
Dingwall, 2007). The application of informed consent to qualitative case studies is 
usually more complex than in quantitative experiments, since quantifying possible risks 
in qualitative studies may be extremely problematic (Murphy & Dingwall, 2007). 
Simultaneously, the nature of the risks inherent in qualitative research differs greatly 
from that in experimental and quasi-experimental studies, and is limited mainly to 
psychological and social hann (Murphy & Dingwall, 2007). The researcher assumes that 
many students may fear that their participation will diminish their chances to obtain good 
grades. Some participants may not be willing to disclose the personal data pertaining to 
their self-efficacy and comprehension levels during the course. Therefore, the goal of the 
informed consent is to ensure that (a) students understand the nature of the study, its 
goals and possible outcomes, and (b) students realize the risk-free implications of their 
participation in the project and the zero effects of such participation on their grades. All 
participants will be provided with a detailed description of the study, its goals, processes 
and anticipated results. Any questions pertaining to the study will be encouraged and 
welcome. By signing the informed consent fonn, the participants will confirm their 
agreement to participate in the study voluntarily, based on the fullest infonnation about 
the study provided by the researcher. 
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Summary 


The design for this study consisted of 20 participants in a graphics design course 
entitled, Digital Imaging. The course ran for five weeks. In studying the differences 
between sampling, this study reflected the method of random sampling. According to 
Creswell (2008), this procedure would allow the researcher to randomly assign 
individuals from both classes to separate groups. Cohen (2006) defined random sampling 
as “a systematic process of selecting subjects or units for examination and analysis that 
does not take contextual or local features into account” (para. 1). 

A faculty member from the School of Education (SoE) from the participating 
university selected the students to be used within the study. Participants were chosen 
from each class on a random basis. Student’s names were printed and cut up and placed 
into a hat. The first 20 names selected were approached to participate within this study on 
a volunteer basis. If a student opted out of the study, another name was chosen to help fill 
the gap. Ensuring the 20 participants would represent one design course and therefore be 
a successful representation of that course. 

Once the SoE had successfully identified the participants, the researcher was 
notified by email. The SoE sent out surveys during weeks one, three and five and also 
received the participant’s responses. The surveys responded to the research questions 
posed for this study. Once the SoE had received all the results from both groups, two 
separate documents were provided to the researcher. The first document was an email 
with all responses provided by the participants and the second email provided the 
document with responses from the control group. There were no identifying marks on 
either survey that would identify either assigned within the perspective groups. By the 


84 



SoE choosing the participants and receiving their responses to the survey eliminated any 
researcher bias within the study. 

The following ethical issues were also be considered: 

• The study sampled enough participants that equaled only one design 
course, therefore may not be representative of all undergraduate design 
courses; 

• Participants may opt out of the research within the five weeks. Depending 
on where a participant chose to opt out of the program would determine if 
an alternate would be assigned to take their place. 

• Opinions expressed by the 20 participants may not be representative of 
opinions within different design courses within the associate’s degree 
program for Graphic Design. 

• Surveys are typically subject to bias when translating the data. 

All these issues had to be considered when the results were interpreted and 
analyzed. The proposed research design enabled the researcher to minimize the risks of 
personal bias, avoid discrimination, deception and confidentiality breaches. 
Simultaneously, some ethical issues, including generalization and population 
representativeness could be successfully resolved in future research. 
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CHAPTER 4. DATA COLLECTION AND ANALYSIS 


Introduction 

As stated in the Statement of the Problem of this study, portfolios have become a 
popular instrument of assessing student learning in a more traditional role of education, 
but not so within distance education. Distance as the distinguishing feature of distance 
education has far-reaching implications for assessment and learning processes (Zawacki- 
Richter et ah, 2011). Distance also necessitates the development of new technologies and 
media that, in turn, distinguish distance education from all other forms of learning 
(Zawacki-Richter et ah, 2011). Distance education by nature implies the sense of 
isolation in students, the lack of face-to-face interactions and delayed instructor feedback 
(Zawacki-Richter et ah, 2011). Apparently, not all students feel satisfied with the absence 
of direct interactions with the instructor, and even the presence of sophisticated 
technologies cannot reduce this relationship gap. Nevertheless, the number of students 
taking online courses rapidly increases: the line between traditional on-campus and 
distance students is fading (Venable, 2010). 

The purpose of this comparative case study was to explore how and to what extent 
self-efficacy (cognitive learning), and levels of comprehension (constructivism), 
impacted students within higher education while creating a portfolio for the distance 
(online) learning environment. In the analysis of portfolios as an instrument of 
assessment, qualitative case studies and similar qualitative methods are believed to be 
more appropriate than quantitative frameworks. Chau and Cheng (2010) further support 
this opinion, by saying that, “understanding the e-portfolio experiences of participants 
and seeking to identify ways in which such experiences are understood and perceived by 
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participants” mandates the use of qualitative approaches (p.935). The reason is that 
qualitative research paradigms substantiate and emphasize the intricacy of human 
realities expressed in assessment portfolios (Chau & Cheng, 2010). Qualitative 
comparisons are of particular use in contexts where the main themes in participants’ 
reflections have to be examined (Lo, 2010). This case study is to analyze the 
effectiveness of e-portfolios as an assessment tool for an online, graphic design course; 
and to evaluate the effectiveness of comprehension of its process. This chapter presents 
both the comparative findings while presenting information between the participants 
within this study and the control group within this study that address and respond to each 
of the research questions designed for this study: 

1 . How do portfolios impact students in an online graphic design course? 

2. How do portfolios impact students’ perceptions of self-efficacy? 

3. How do portfolios impact students’ comprehension? 

The remaining portions of this chapter will provide information regarding 
participant recruitment, describe the process of selecting the participants, sending surveys 
to the participants and control group, how that data was collected; and finally, how the 
participant’s portfolios were collected with a final assessment given to the participants by 
utilizing a rubric designed for this study. Completing this chapter will be the findings of 
this comparative case study based on the data collected, coded and analyzed. 

Description of the Data 

This comparative case study concerned the students of two Digital Imaging 
courses at a university located in central Florida. Digital Imaging is a course required for 
undergraduate students earning an associate’s degree in graphic design. The university 
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has both, on-ground campuses, as well as distance learning (online) classrooms. These 
specific courses were taught through the online portion of the university and the 
participants were selected solely attending the distance or online learning environment. 

Involved within this study were faculty and staff to ensure the validity of student 
anonymity and biases that could present themselves during this study. Involved were the 
department chair, Research Assistant 1 (RA1), Research Assistant 2 (RA2), and the Dean 
of Education. The department chair was key in assigning students to the two classrooms 
based on how this course aligned within the programming of their major, for students 
entering the Digital Imaging course; RA1 volunteered her time and services to help with 
the random sampling of participants, sending and collecting surveys, coding surveys and 
e-portfolios; and finally, RA2 was responsible for collecting the participant’s final e- 
portfolios and provided a final grade utilizing the rubric designed for this specific study 
and to validate the reliability of the rubric. The instructor for these courses is also the 
researcher; therefore justifying the need for faculty to assist within this research process. 
Included in this study were two online courses consisting of 20 enrolled students within 
each course, involving a total of 40 students. With the demographic being an online 
course within the graphic design, undergraduate degree program, each student enrolled 
within the course came into the course with the same basic skill requirements and 
knowledge of the software being taught for a period of five weeks. The goal for this study 
was to be able to obtain 20 participants who would create e-portfolios, which was 
representative of one online course. Gender, age, or cultures were not a condition of this 
study as these are common traits found within a quantitative or mixed methods study. 
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Field Test 


A field test was conducted by submitting an official Institutional Review Board 
(IRB) request to five experts within the graphic design field who were also experienced 
in working within the online distance education and learning environment. Requests were 
sent to experts to examine the following instruments; 1) Self-efficacy for the control 
group (Appendix A); 2) Self-efficacy survey for participants (Appendix B); 3) Rubric 
that measures the comprehension of concepts in design (Appendix C); and 4) Rubric that 
measures comprehension of skills in design (Appendix D). 

The first field test was conducted for both rubrics for this study. The purpose of 
these rubrics for this study was to create a rubric that measured student’s comprehension 
design skills (Appendix D), as well as student’s comprehension levels of design concepts 
(Appendix C). Also submitted to five experts within the design field were the survey 
questions that were used to measure the impact of creating a portfolio within a graphic 
design course (Appendix B), as well as measuring a student’s self-efficacy (Appendix A), 
for the control group not involved in the process of creating a portfolio. The goal of these 
surveys and rubrics was to respond to each of the following research questions; 1) How 
do portfolios impact students in an online, graphic design course; 2) How do portfolios 
impact students’ perception of self-efficacy; and 3) How do portfolios impact student’s 
comprehension? 

Rubric for Comprehension of the skills of design 

Before sending out the rubrics to various experts, the researcher provided a grade 
for one of the designs within the Digital Imaging course. This course is a beginning 
Digital Imaging course that requires students to use Adobe Photoshop to create a collage- 
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based design over the course of five weeks. Based on the rubric provided for this task, the 
researcher graded this project as a five, or a letter grade of an “A”. The rubric being used 
for this task is shown in Appendix D that provides the comprehension grade of the 
students’ design skills for this specific course. Out of the five experts who turned in 
grades for this project, four agreed that the assignment provided, fit the requirements of 
the grade of 5 according to the rubric and the other provided a grade of 4. The expert that 
graded the assignment as a 4 stated, “Comprehension of Skills in Design as students 
completed all the required elements but needed to do more in applying required 
techniques. It was difficult to grade the assignment without having detailed knowledge of 
the processes involved”. No specific changes were requested nor provided from any of 
the experts as to how this should be or could be modified. 

Rubric for comprehension of the skills of concepts 

This particular rubric was designed to measure the comprehension of design 
concepts within an online graphic design course for an undergraduate degree. Students 
are required to post two responses to the discussion boards each week. Because this 
assignment is discussion based and being able to use tenninology of a graphic designer, 
no samples were sent to perspective experts. Instead, the experts were asked to evaluate 
the rubric that measures “Comprehension of Concepts” (Appendix C). The experts 
participating within this field test stated that no changes were needed. 

Summary of experts ’findings for rubrics 

Five experts within the field of higher education, online education and graphic 
design each provided a summary for each of the rubrics which measured both; 
comprehension of skills and comprehension of concepts within an online graphic design 


90 



course (Digital Imaging) for the undergraduate program required for students in order to 
earn an Associate’s of Fine Arts degree. Each of the experts’ feedback was positive in 
nature, but indicated a difference in tenses used within each rubric. Each expert 
recommended that all wording reflect “past tense” as students’ projects would have 
already been completed prior to using the appropriate rubrics to measure either the 
students’ comprehension of skills or comprehension of concepts for the specific 
assignments. The rubrics were revised per the experts’ recommendations. 

Participant ’s self-efficacy survey 

For this study, a field test for a “Participant Self-Efficacy Survey” (Appendix A) 
was created to test for validity and reliability of the research questions for this 
comparative case study. This open-ended survey was designed to measure the 
comprehension of the process of creating a portfolio within an online graphic design 
course, as well as measuring the varying levels of self-efficacy during the portfolio 
creation process during a five-week course. As a result, an official IRB request was sent 
to five experts in the field of Graphic Design, higher education and online courses. Of 
those five requests, the five experts responded favorably for this study, thereby validating 
this instrument as responding to the research questions, allowing the researcher to utilize 
a comparative case study. The experts participating within this field test requested the 
following changes: 

Expert 1 - Ql) Wording should be changed to read; Please include an estimated 
time spent weekly on task activities; Q5) Considering your current level of experience, 
confidence, problem-solving abilities and level of interaction with your instructor when 
issues or questions arose, please describe your level of expertise in creating portfolios 
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prior to this class. Please be explicit; Q6) Please include a detailed rationale for your 
reasoning; and Q12) How often did you need additional assistance from your instructor, a 
tutor or a classmate? Changes were made according to the experts’ requests. 

Control group self-efficacy survey 

For this study, a field test for a “Control Group Self-Efficacy Survey” (Appendix 
B) was created to test and validity and reliability of research questions for this 
comparative case study. This open-ended survey was designed to measure the self- 
efficacy of the control group for this study. The control group’s survey differs in the 
respect that members of the group will not be completing a portfolio. The students’ self- 
efficacy will however, be measured and compared to the participant’s level of self- 
efficacy. The control group will receive exactly the same summative feedback for the 
discussion board assignments and design projects, and will also receive the same lectures 
each week as the participants of this study. The only difference between the two groups 
will be that the control group will not be involved within the portfolio creation process, 
therefore, the open-ended questions needed utilized for the control group ought to be 
revised. As a result, an official IRB request was sent to seven experts in the field of 
Graphic Design, higher education and online courses. Of those seven requests, five 
experts responded favorably for this study, thereby validating this instrument as 
responding to the research questions, allowing the researcher to utilize a comparative 
case study. The experts participating within this field test requested minor changes, but 
none that would change the overall meaning of the question. 
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Instruments 


For this comparative case study, no certified instruments were available for use. 
Therefore, for this comparative case study, four instruments have been created by the 
researcher; 1) Open-ended survey for the control group (Appendix A); 2) Open-ended 
survey for the participants (Appendix B); 3) Rubrics that will measure comprehension of 
design skills (Appendix C); and 4) Rubrics that measures the comprehension of design 
concepts (Appendix D). A field test was conducted by sending the prescribed instruments 
to the five experts within the graphic design field in order for the experts to assess the 
strengths and weaknesses of the prescribed instruments. Field tests were completed and 
approved by the Institutional Review Board (IRB) prior to the data collection process. 

Prior to the beginning of the design course, prospective students of the Digital 
Imaging undergraduate graphic design course were sent welcome emails. Within the 
email was an introduction about the research, outlining the importance of students 
responding to any emails from RA 1 . The students were notified that participating in 
creating a portfolio for the design course was to be absolutely voluntary and students 
would have the option to opt out if they were selected as participants. 

Both the control group and the participants were graded using the same rubrics for 
this course (Appendices C-D), which measured each student’s comprehension of the 
concepts of design, as well as students’ comprehension of the skills required and learned 
throughout the Digital Imaging course. The instructor and researcher for this case study 
taught both courses simultaneously. 

A field test was conducted by sending the required instruments for this study to 
the School of Education (SoE) of the university. Both faculty and staff assessed the 
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strengths of the rubrics and the questionnaires to ensure its reliability for this research and 
evaluated these instruments. The need for an independent assessment of rubrics is 
justified by the controversy surrounding the validity and reliability of rubrics as a form of 
learning assessment (Rezaei & Lovom, 2010; Jonsson & Svingby, 2007). The field test 
utilized an assignment created by another instructor who had previously taught the same 
course. The assignment was graded per the rubrics provided for comprehension of design 
concepts and comprehension of skills of design (Appendices C-D). When completed, the 
assignment was forwarded to the SoE for further evaluation and assessment against the 
rubrics provided. Likewise, the same experts evaluated and assessed the strengths and 
weaknesses of the open-ended surveys designed to provide data for the comparative case 
study (Appendices A-B). Jarobe (2009) stated, “Field tests are often recommended to 
assess the strength of research questions/hypothesis or test the appropriateness of certain 
data-gathering protocols (e.g., informed consent procedures), instruments/tools (e.g., item 
content) and data analysis procedures” (para. 2). Once experts had reviewed the 
instruments, the IRB process commenced. 

Data Collection 

The following instruments were used to collect the data for this comparative case 
study: participant self-efficacy survey (Appendix A), control group self-efficacy survey 
(Appendix B), and rubrics to measure comprehension of design concepts (Appendix C), 
and comprehension of design skills (Appendix D). All instruments of data collection 
were field-tested and approved before being implemented into the research project. 

Before research and data collection started for this study, the instructor informed all 
students entering into the two Digital Imaging courses of the research that was to take 
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place during the course, by sending all students a welcome email. Emphasized within the 
email was the assurance that each student’s instructor would not know who was 
participating or who was not participating in the portfolio creation process. Also outlined 
within the email was the importance of students forwarding all questions or issues about 
the research, the research study and what would be involved once class began to the 
instructor via email, the classroom portal instant message system or by phone. Once the 
classes began, students could no longer ask the instructor questions in order to prevent 
potential biases and knowledge of who was or was not participating within the research 
portion of this project. 

On day one of the course, the RA1 placed all the student names from the two 
different classes into a hat. The instructor selected 20 students to potentially be the 
participants for the study. Once chosen, the students were issued an informed consent 
form outlining the nature of the study and requesting the students’ permissions to be 
included as participants within the study. If a student declined the request, RA1 selected 
another student and the same process was repeated until RA1 received a total of 20, 
signed informed consent forms. 

With the participants chosen randomly for this study, RA1 sent the remaining 20 
students an email infonning them that their names were not chosen to participate within 
the portfolio creation process, but explaining that the students will still participate within 
the survey process (Appendix A). Attached to the email was an informed consent fonn 
for the control group, which outlined the expectations of the study and the role and 
responsibilities of the proposed control group for this study. All forms were returned and 
signed and fded on the computer of RA1, and protected by a personal password to ensure 
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student anonymity. At the end of weeks one, three and five, RA1 sent surveys to both the 
participants and control group, requesting a 24-hour time frame for the selected group 
members to complete and return. When RA1 received the emails with the surveys 
attached from each person, the RA1 coded each survey by providing a randomized 
numbering system and created an excel spreadsheet for the purposes of the research, to 
track the surveys each week. Ensuring there was no identifiable information of the 
participants within the surveys, RA 1 sent the surveys to the instructor who was also the 
researcher for this study. Once the instructor received these surveys from both; the 
participant group and the control group, she entered the data received into SurveyMonkey 
and HyperResearch. SurveyMonkey was used to code and analyze any/all data for the 
survey questions that were not open-ended. HyperResearch is MAC based software for 
qualitative studies. All open-ended questions were entered into this software, coded with 
keywords from the participants’ surveys and analyzed. 

Participants differed from the control group, only because the participants created 
a portfolio during all five weeks of the design work. Each week during the instructor’s 
lecture, the instructor demonstrated the processes necessary to create a portfolio for the 
participants’ weekly design project. For example, during week one, the participants 
would save their design work with the requirements stated for that week’s design project. 
Upon completion of week one, the participant will have saved the work as a multi-page 
Portable Document Format File (PDF). Each week thereafter, the participants would 
follow the same format for weeks two through five and add their progress to the week 
prior. 
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Upon completion of the design course, the participants sent their final portfolios 
to RA1 who then coded the assignments to coincide with the identifier assigned in week 
one. In turn, RA1 sent the portfolios to RA2, who manually went through each project to 
detennine the grading for each of the portfolios. The grading was based on the rubric that 
measured the participant’s skill of design (Appendix D). RA2 provided justification for 
each of the grades provided. RA2 held onto the grades until the instructor provided the 
final grades - within the participants’ grade books - to the university. Once the instructor 
posted her final grades, RA2 forwarded his results of his grades to her and the instructor 
then compared those results to her final grades. 

Non-Participants - Group B was used as the control group. Because the instructor 
was not to be privy to the identity of either group within this study, Group B received the 
same information and feedback during class lectures as Group A. The only difference 
were the participants’ (Group A), created a portfolio and the non-participants (Group B) 
did not. 

In both classes, participants and non-participants completed the discussion board 
task (part one) by day three of each week, with each final discussion due by day seven. 
Also due by day seven were the participants’ and non-participants’ required design 
project for the week. Fonnative feedback was provided to both groups. The instructor 
also provided two, one-hour live lectures via Adobe Connect™ , which is currently 
included within students’ classrooms. Participants and non-participants also received a 
weekly grade entered into their individual grade book for both - their discussion board 
projects and their design projects based on the rubrics designed, tested and validated for 
this study (Appendices C and D). Besides a letter grade, both groups were also provided 
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written feedback, which included positive feedback, as well as critical written feedback 
that provided recommendations on how to further enhance their discussions or their 
individual designs. Other tools available to both groups were the opportunity to email the 
instructor or contact the instructor by phone if the participants or the control group 
required any clarification or help with any aspect of the assignments. 

According to Stoddart (2006), a combination of multiple methods of data analysis 
“will ensure the validity of the study” (McMillan & Schumacher, 2001 as cited in 
Stoddart, 2006, p. 39), such as the use of SurveyMonkey - that was used to analyze the 
data entered into its database by the researcher. This process was started by RA1 who 
coded each of the surveys by assigning numbers to both the control group and the 
participants. The open-ended surveys for both - the participants and the control group, 
each contained 10 questions requiring these groups to respond with the Likert Scale, 
given five points, plus a strongly agree or strongly disagree option. Data collected from 
both groups were then entered into Survey Monkey in order to provide a potential theme 
that responded to learner-instructor interactions and perceived self-efficacy each week. 

Once the data was entered into SurveyMonkey, charts were generated based on 
participants’ responses. Once the information had been recorded from the database for 
the comparative case study, it was implied that the results would successfully respond to 
the research questions, as well as provide any insight to new, emerging themes that would 
be provided through the actual analysis of the perfonned data collection. 

The final six questions within the surveys were open-ended responses. These 
responses from the both groups were then entered into HyperResearch software, which 
was a qualitative data analysis program for Macintosh users that allowed the researcher to 
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“code, retrieve, build theories and conduct analyses of qualitative data” (Stoddart, p. 39). 
From the narrative responses provided by the participants and control group, common 
words were chosen as keywords and placed into HyperResearch. The coded words 
reflected common terminology used within the participants’ responses and then applied 
for each of the questions. Once consistent responses were provided for these questions, 
all data was entered into the database and coded, producing common themes from the 
surveys and perhaps provided new themes for future research. Baker (2003) talked about 
questionnaires as being, “the principle means used for collecting data by means of a 
survey of a designated population or sample in which the researcher is interested” (p. 
343). 

Data Analysis 

In analyzing the data, the goal for this study was to provide data demonstrating to 
students within online graphic design courses the importance of creating a portfolio of 
students’ design work through each of the design courses students are required to attend 
in order to achieve an undergraduate’s degree in graphic design. Along with the 
importance of creating these portfolios the data hopes to prove through concept analysis, 
the improvement or increased levels of the participants’ self-efficacy. Zulkosky (2009) 
stated concepts being defined as, “a word or phrase that summarizes ideas, observations 
and experiences” (p. 93). By creating a portfolio, the participants were able to view their 
successes within the work provided for the portfolio each week which allowed for that 
“mental image that can facilitate communication about and understanding of phenomena” 
(Fawcett, 2005, p. 4, as cited in Zulkosky, 2009). Continuing along these lines of mental 
images (or participants viewing individual progress each week), Mezirow’s 
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Transformative Learning Theory begins its process of using the “frame of mind” (1987, 
p. 6) to transfer that mental image into personal success, thereby, increasing personal 
levels of comprehension and self-efficacy. 

Comparisons and analysis was used to determine both groups’ self-efficacy and 
comprehension levels throughout the participants’ and the non-participants’ course. 
Shulman (2003) discussed five benefits in the use of portfolios as being; 1) portfolios 
pennit tracking of longer periods of teaching verses a single observation; 2) portfolios 
can bridge gaps between the process and the product; 3) “portfolios institutionalize norms 
of collaboration, reflection and discussion”; 4) a portfolio provides structure; and 5) a 
portfolio shifts the responsibility of teaching - to learning - back to the participant 
(constructivist learning) (as cited in Lombardi, 2008, p. 9). 

The data collected and analyzed for this study included only online students 
within the graphic design course within this university. The researcher for this study was 
assigned two Digital Imaging course consisting of approximately 40 online students 
within the associate’s degree program in graphic design. RA1 randomly chose 20 
students between the two classrooms, collected data and provided that data to the 
researcher. Ultimately, the count for the participants and control groups both contained 
20 students, which was representative of one online classroom. Hybrid students - 
students assigned to an on-ground campus, were eliminated from this research and for the 
purpose of this study. The following portion of this chapter provides the data collected 
and how this data related to each of the research questions. 
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Ql. How do portfolios impact students in an online graphic design course? 

In analyzing the data collected from participants and the control group, this study 
demonstrated the required responses to the research questions. The first sub-question to 
this study: How do portfolios impact students in an online design course? In week one, 
the researcher asked both the participants and the control group, how many hours they 
spent each week within the online classroom. The success of a student within an online 
course is demonstrated through how much time is spent reading the required text for the 
course, reviewing course material and the time spent in creating the design work each 
week in preparation for the work to be evaluated. The assumption for a student with no 
experience within the assigned software program would be an average of six to ten hours 
per week for each week of the students’ studies. If the course is harder than originally 
anticipated, the student would require more time within the classroom and involved in 
research, and reviewing the live chats that were recorded for the students’ convenience. 
Table 1 reflects the actual time as recorded by participants during weeks one, three and 
five of the design course. 

Table 1. Time Spent Within The Online Design Course 


Participants 



< 5 hours 

6-10 hours 

11-15 hours 

1 6-20 hours 

> 20 hours 

Week 1 


33% 

16% 

42% 

9% 

Week 3 

10% 

57% 


33% 


Week 5 

20% 

60% 


20% 



Note: < = less than, > = greater than 

Table 1 reflects the difference in time that participants spent within the online 
design course for an associate’s graphic design degree. Originally, 42% of the 
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participants spent between 16 and 20 hours within the design course. As each week 
progressed, there is a steady decline of how much time was required to complete 
individual tasks and to create the portfolio required for this study. Table 2 reflects the 
actual time as recorded by the control group during weeks one, three, and five of the 
students’ online design course. 

Table 2. Time Spent Within The Online Design Course 


Control Group 


< 5 hours 

6-10 hours 

11-15 hours 

1 6-20 hours 

> 20 hours 

Week 1 


36% 

32% 

15% 

15% 

Week 3 


30% 

42% 

28% 


Week 5 


33% 

33% 

34% 



Note: < = less than, > = greater than 


Table 2 reflects the control group spends online within the given classroom. The 
control group was provided the same instruction, provided the same classroom materials 
and the same opportunities of contacting personal instructors for additional help within 
the course. In week one, the majority of the control group spent less time within the 
course reflecting 36% of the control group only spending between six and 10 hours. By 
week five, the number shifts to show that 34% of the control group spent between 16 and 
20 hours within the online course. 

Data collection for this study included questions within the surveys to reflect 
interaction between the participants and instructor, as well as the instructor and the 
control group. The assumption prior to collecting this data was that the participants would 
interact more with the instructor during the beginning weeks of the design course and less 
as the class progressed. With the basic understanding of creating a portfolio, being able to 
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see personal work progress each week would improve participants’ perceived self- 
efficacy which responds to the second and third research question for this study which 
are: 1) How do portfolios impact students’ perceptions of self-efficacy and 2) How do 
portfolios impact students’ comprehension? A Likert scale was created for the first 10 
questions within the self-efficacy survey for the participants (Appendix B), and the self- 
efficacy survey for the control group (Appendix A). Tables 3 and 4 reflect the actual 
responses provided by each group. 

Table 3 states that during week one, participants had fewer interactions with the 
instructor than weeks three and five. Also stated is the increased interaction through 
email and electronic media during weeks three and five. During week one, the 
participants did not interact by asking questions or asking for help. For the ones that did 
require assistance, the instructor responded within a timely fashion. For Q5 in week one, 
it shows a lack of fonnative feedback from the instructor at this point within the design 
course. No assignments at this point were turned into the classroom for comments or 
grading during this period of week one, reflecting an expected response. During weeks 
three and five, participants interacted more with the instructor by asking questions, 
sending emails or requiring extra tutoring. 
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Table 3. Participant - Instructor Interaction 






Participants 





(Strongly disagree 1 2 

3 4 5 N/A 

Strongly agree) 




Numerous 

Interactions 

Utilized 
electronic 
means to 
contact 
instructor 

Instructor 
responded 
within a timely 
fashion 

Participant 
responded to 
instructor’s 
response 

Participant 
received both 
positive and 
informative 
feedback 

Week 1 

1 

2 

18% 

8% 


7% 



3 

15% 

17% 

7% 

17% 

17% 


4 

25% 

7% 

18% 

8% 

16% 


5 


50% 

58% 

50% 

16% 


N/A 

42% 

18% 

16% 

18% 

50% 

Week 3 

1 

2 


22% 


10% 



3 

10% 






4 

55% 

15% 


12% 



5 

22% 

58% 

78% 

78% 

100% 


N/A 

13% 

5% 

22% 



Week 5 

1 

2 







3 







4 

40% 

40% 



20% 


5 

60% 

60% 

100% 

80% 

80% 


N/A 




20% 




Qi 

Q2 

Q3 

Q4 

Q5 


Note: Q = question per the survey 


According to the data, the instructor responded within a timely fashion and 
provided both positive and informative feedback throughout the rest of the course. Table 
4 was created to reflect the same questions as the participants but the word portfolio was 
removed and replaced with the word design to reflect an accurate assessment of the data. 
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Table 4. Control Group - Instructor Interaction 


Control Group 

(Strongly disagree 1 2 3 4 5 N/A Strongly agree) 



Numerous 

Interactions 

Utilized 
electronic 
means to 
contact 
instructor 

Instructor 
responded 
within a timely 
fashion 

Participant 
responded to 
instructor’s 
response 

Participant 
received both 
positive and 
informative 
feedback 

Week 1 






1 




18% 


2 



12% 


18% 

3 

18% 

35% 




4 

18% 

15% 



18% 

5 

33% 

50% 

88% 

82% 

50% 

N/A 

31% 




14% 

Week 3 

1 






1 

2 






3 

24% 


18% 



4 

28% 

25% 


42% 

42% 

5 

48% 

75% 

82% 

48% 

58% 

N/A 






Week 5 

1 






1 

2 






3 


33% 

33% 



4 

50% 



27% 


5 

50% 

67% 

67% 

73% 

100% 

N/A 







Qi 

Q2 

Q3 

Q4 

Q5 


Note: Q = question per the survey 


Table 4 indicates that during week one, the interaction between the control group 
and the instructor was mainly through electronic means (email) or feedback within the 
control group’s grade book. Weeks three and five shows an increased need for the 
instructor interaction with a high satisfaction level from the control group regarding the 
instructor’s response time and feedback. 
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Q2. How do portfolios impact students ’perceptions of self-efficacy? 

Students entering into an online graphic design course have some sense of self- 
efficacy pertaining to the use of software for the specific design course. The requirement 
for students entering into the Digital Imaging course had no pre-requisites attached to the 
course and was placed into this course based on their placement within their 
undergraduate degree program. Students are perceived as going into the course with little 
or no experience within this specific course. Table 5 is a recorded view from the 
participants while entering into the Digital Imaging course - by entirety using the Likert 
Scale. During each week of the course, the participants created a portfolio that 
demonstrated the process of the design work on a continual basis. 

Table 5 reflects the perceived self-efficacy of the participants. During week one, 
58% of the participants had not created portfolios prior to the course, with 10% stating 
their personal confidence level showing as being confident enough to be able to problem 
solve. The ability to problem solve was stated by 25% of the participants. Also in week 
one, 3% of the participants would recommend this process as part of all the students’ 
design courses. In week three, the numbers tend to shift to the center of the Likert Scale 
and in week five, those numbers shift to improved self-efficacy, problem solving and 
recommending the process of creating a portfolio for all the design courses. 
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Table 5. Participant - Perceived Self-Efficacy 


Participants 

(Strongly disagree 1 2 3 4 5 N/A Strongly agree) 



I have created 
portfolios prior 
to this course 

I felt confident 
I would enjoy 
the process of 
creating a 
portfolio 

During the 
process, I had 
to contact the 
instructor with 
questions 

I felt confident 
enough to 
problem solve 
on my own 

I would 
recommend 
this process for 
all my design 
courses 

Week 1 






1 

58% 

3% 

25% 

10% 

3% 

2 

2% 

5% 

15% 

15% 


3 

2% 

44% 

25% 

25% 

23% 

4 


15% 

10% 

25% 

25% 

5 


33% 

8% 

16% 

42% 

N/A 

38% 


17% 

9% 

7% 

Week 3 






1 

38% 

12% 

10% 

8% 


2 


12% 




3 


33% 

25% 

55% 


4 

12% 

12% 


15% 

21% 

5 

15% 

33% 

33% 

22% 

73% 

N/A 

35% 


12% 


6% 

Week 5 






1 


40% 

20% 



2 






3 


20% 


20% 


4 




40% 

20% 

5 

5% 

40% 

60% 

40% 

80% 

N/A 

80% 


20% 




Q6 

Q7 

Q8 

Q9 

Q10 


Note: Q = question per the survey 


Table 6 reflects the views of the control group. The word portfolio was replaced 
with “design course” The data shows that 82% of the control group had never used the 
required software when entering into their course and 33% felt confident enough to 
problem solve the design problem on their own during week one. By week five however, 
those numbers shifted to 75% of the control group feeling confident to problem solve 
their design problem. 
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Table 6. Control Group - Perceived Self-Efficacy 


Control Group 

(Strongly disagree 1 2 3 4 5 N/A Strongly agree) 



I have used this 
software prior 
to this course 

I felt confident 
that I would 
enjoy creating 
my design 
projects 

During the 
process, I had 
to contact my 
instructor with 
questions 

I felt confident 
enough to 
problem solve 
on my own 

I would 
recommend 
this process for 
all my design 
courses 

Week 1 






1 

82% 

24% 


17% 


2 

18% 

13% 




3 


14% 

19% 

33% 

19% 

4 


16% 

33% 

17% 

15% 

5 


33% 

32% 

33% 

33% 

N/A 



16% 


33% 

Week 3 






1 

50% 





2 

14% 


13% 

15% 


3 


35% 

23% 

42% 

9% 

4 

22% 

23% 

19% 

15% 

33% 

5 


42% 

45% 

28% 

58% 

N/A 

14% 





Week 5 






1 

100% 

35% 




2 


33% 




3 



25% 



4 




25% 

30% 

5 


32% 

75% 

75% 

70% 

N/A 







Q6 

Q7 

Q8 

Q9 

Q10 


Note: Q = question per the survey 


HyperResearch is a qualitative data analysis tool that allows its users to code data, 
generates reports and theories based on the data. For this study, HyperResearch was used 
to analyze open-ended questions of both, the participants and the control group. The 
participants within each group were given different surveys probing the final series of 
questions. Keywords were provided by participants within each group, coded in 
HyperResearch and analyzed. Table 7 reflects the views of the participants and the 
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perceived self-efficacy of these participants while creating a portfolio and its usefulness 
within a design course. 

Table 7. Participants - Changes in Perceived Self-Efficacy 


Participants 



How did 
creating a 
portfolio 
increase 
your level of 
competence? 

Did you 
notice any 
benefit of 
creating a 
portfolio? 

Would you 
use a 

portfolio as 
a learning 
tool? 

Would you 
use the 
portfolio as 
an 

assessment 

tool? 

Do you 
feel more 
confident 
in 

repeating 
the process 
for future 
classes? 

How has 
your 

confidence 
level 
changed 
during this 
process? 

Week 1 







Great Benefit 

11 

10 

13 

15 

9 

7 

No Change 

3 

6 

2 

0 

3 

0 

No opinion 

6 

4 

5 

5 

3 

13 

Week 3 







Great Benefit 

15 

19 

17 

15 

17 

11 

No Change 

1 

1 



1 

2 

No opinion 

4 

4 

3 

5 

3 

7 

Week 5 







Great Benefit 

19 

15 

17 

18 

18 

19 

No Change 


4 



2 

1 

No opinion 

1 

1 

3 

2 




Oil 

Q12 

Q13 

Q14 

Q15 

Q16 


Note: Q = question per the survey 


Q3. How do portfolios impact students ’ comprehension? 

Table 8 reflects participants' and control's final grades and comprehension grades. 
These grades were determined based on control and participant scores for rubrics 
designed to measure comprehension of design concepts and comprehension of design 
skills. The instructor computed participants' and control's final grades by using the rubric 
designed selected for this study. 50% and 25% of the participants and control groups 
respectively scored an “A” for their final grades. Also 45% and 5% of the participants 
had B and C final grades respectively with 25% and 10% of the control group having B 
and C final grades. 40% of the control group had F final grades. While 85% and 15% of 
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the participants had A and B comprehension grades respectively, 50% and 15% of the 
control had A and B comprehension grades respectively. 15 % and 10% of the control 
group scored C and D comprehension grades respectively and a 5% rate each for E and F 
comprehension grades was recorded. 

Table 8: Final Skills and Comprehension Grades 

Grade 

A B C D E F 

Final Grade 


Participant 

50% 

45% 

5% 


Control 

25% 

25% 

10% 

40% 

Comprehension Grade 





Participants 

85% 

15% 



Control 

50% 

15% 

15% 10% 5% 

5% 


Research Analysis 

An effective analysis of the primary data provides evidence showing the possible 
significance of portfolio creation by online graphic design students. The primary research 
provided the participants' and control's data related to different indicators or influencing 
driving factors of students' performance. This section is a comparative analysis of the 
results of the participants and control group with respect to these educational 
performance indicators. The primary research utilized the following indicators: time 
spent within the online design course, instructor interaction, participant perceived self- 
efficacy, level of comprehension, and final rubric grade. 

One of the major factors determining a student's success within an online learning 
environment is how much time the student spends in reading the required course text, 
reviewing all materials and completing the tasks weekly in preparation for the tasks to be 
evaluated. A shorter completion time indicates an improvement in the student's 
performance level in an online course. Therefore, comparing the difference between the 
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completion time for students participating in portfolio creation and the control group 
(non-participants) will contribute to identifying the possible influence of portfolio 
creation on the perfonnance of graphic design students. Tables 1 and 2 indicate the time 
spent by the participants and control in weeks 1, 3 and 5 respectively. In the first week, 
more than 50% of the participants group spent over 15 hours in within the online design 
course while only 30% of the control group spent up more than 15 hours for the online 
graphic design course. This is an indication that in the first week of commencing 
portfolio creation, the control group actually performed better than those participating in 
portfolio creation in terms of time used to complete the online course graphic design 
activities. 

There is a change in the time spent for the course completion indicated for 
participants and the control group by Week 3. Tables 1 and 2 show that at Week 3, 
participants 67% of the students participating in portfolio creation perfonned below the 
average time of completion (6-10hours), with 10% of these students actually completing 
the graphic design tasks in less than 5 hours. However, the percentage of control 
performing individual tasks in at the average completion time rate was 30% (a reduction 
of 6% from week 1) with not student completing individual graphic design tasks for the 
week in less than 5 hours. Therefore, by Week 3, the amount of students participating in 
portfolio creation who completed the graphic design tasks within average completion 
time was over 100% above that of the non-participating students. 

By week 3, there was a further increase in the difference in completion time 
between the participants and the control group. 80% of the students perfonning portfolio 
management successfully completed individual tasks within the average time (with 20% 
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completing individual tasks in less than 5 hours). The rate of control students completing 
the graphic design tasks within the average completion time (6-10 hours) is 
comparatively significantly low (33%). From the analysis, it is students' participation in 
portfolio creation improves completion time of online graphic tasks and may therefore 
positively influences the general performance students in online graphic tasks. 

Instructor interaction is also a significant factor influencing and indicating the 
students’ performance (Van de Pol, Volman, & Beishuizen, 2011). The level of 
interaction between the students and instructors as well as between students and other 
students show indicate student engagement, which is another factor shown to indicate 
student’s performance and achievement (Tosh, Light, Fleming, & Haywood, 2005). It 
was necessary to consider the influence of student-teacher interaction as one of the 
indicators influencing students’ performance in the current case owing to the nature of 
the learning environment characteristic with the sample considered. Since the graphic 
design course considered in this research is an online based course, considering the 
effects of the learning method on teacher interaction was necessary; considering the 
effects of portfolio creation on interaction was also necessary. Since distance education 
and learning is naturally associated with the idea of isolation and the absence of face-to- 
face interactions with instructors (Suen, 1996), understanding the implications of 
portfolio assessment on teacher interaction in a distance learning graphic design course 
became necessary. Thus, a comparative analysis of the influence of portfolio 
management on students’ level of interactions with the instructors in the graphic design 
distance learning environment became necessary. Tables 3 and 4 present an overview of 
the outcomes of strategically developed open end questions investigating the level of 
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interaction shared between students participating in portfolio creation and the control 
group (non-participants). 

One of the factors used to investigate the effects of portfolio management on 
student-instructor interaction was the interaction frequency between the students and the 
instructors. From Tables 3 and 4, the outcome of Q1 indicates that from Week 1 to Week 
3, there was an increase in the amount of interactions for both students participating in 
portfolio creation and the control group (non-participants). This indicates that the amount 
of interaction between students and instructors over time is not necessarily dependent on 
students' participation in portfolio creation. Another factor used to consider the effects 
portfolio management of the student-instructor interaction effectiveness was mode of 
interaction; this was done by investigating how often students used mails to communicate 
with instructors. This method has been used in previous research (Van de Pol, Volrnan & 
Beishuizen, 2011). While the results showed a continuous increase in the amount of 
electronic mail communication between participants and instructors from week 1 to week 
3, there was a somewhat constant amount of electronic mail communication between the 
non-participants and instructors at week 1 and week 2, and a reduction in electronic mail 
interaction at week 3. 

There is a considerable difference in instructor response satisfaction rate between 
participants and control. While participants' satisfaction rate increases weekly from week 
1 and peaks at week 3, there is significant reduction in the satisfaction rate of the control 
group. This is an indication of the positive effects of portfolio creation on the online 
graphic design course because participants' perform better at individual tasks and are 
naturally satisfied with instructors' positive responses to these tasks. Distance learners 
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usually tend to have clearer goals and reasons for engaging in a distance course, but the 
degree of responsibility for managing contacts with the instructor varies across 
individuals (Suen, 1996). It is obvious from the students-instructor interaction results of 
that the portfolio creation can influence this degree of responsibility and improve student- 
instructor interaction. These findings are supportive of Mezirow’s theory, suggesting that 
portfolio assessments emphasize educators’ role in learning as helping students to engage 
in reflective thinking and helping the students to redefine personal understandings and 
insights (Conrad, 2008). 

The research study also considers students' perceived self-efficacy as a factor 
influencing or indicating students' performance level. Self-efficacy refers to people’s 
confidence to accomplish tasks or achieve goals (DeTure, 2004) as well as people’s 
ability to judge how well personal learning tasks can be achieved (Saade & Kira, 2009; 
Marsh & Martin, 2011). The importance of self-efficacy on educational achievement and 
accomplishment of learning tasks has increased its position as a student perfonnance 
indicator (Trautwein, Ludke, Nagy & Marsh, 2009), and this is also evident in its 
repeated use in various research studies (e.g. Wahab, 2007 & Wadsworth, et ah, 2007). 
Based on this consideration, it is necessary to compare the participants’ and control (non- 
participants) self-confidence levels in order to identify the influence of portfolio 
assessment on the students’ self-efficacy, and subsequently on the course performance. 

Tables 5 and 6 present the different levels of self-efficacy expressed by the 
participants and control over the weeks. Participants' self-efficacy level, judged by how 
confident the participant felt with regards to enjoying the process of creating a portfolio 
constantly increased from Week 1 to Week 5, while control level of self-efficacy, judged 
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by personal confidence level for project designs reduced over time and was the least at 
the third week. Participants were also more confident to recommend the portfolio 
creating method to be used by other online graphic design students than control (non- 
participants), whom were confident, but exhibited less confidence than the participants. 
The results in Table 7 further indicate that creating a portfolio consistently improves self- 
perception of competence and general student confidence The findings for this particular 
construct is supported by the suggestion that portfolio assessments have the potential to 
drive students’ self-efficacy, since students create an atmosphere of learner-driven 
education; the latter is the foundational ingredient of the constructivist philosophy (Sajadi 
& Khan, 2011). Since higher learner self-efficacy beliefs are fundamental to the 
professional maturation of students (Jones, 2009), creating a portfolio presents as a viable 
method of improving students’ perfonnance. 

After understanding the general effects of how creating a portfolio influences 
students' characteristics such as student-instructor interaction and self-efficacy, it will be 
necessary to identify the effects of this characteristics on students' general comprehension 
levels owing to the interrelationship between these factors (Aries, 2010). Comprehension 
is defined as a student’s capacity to perceive and understand the meanings communicated 
by online instructors and the requirements placed on these instructors in terms of 
graduation (Caldwell, 2008). It is also explained as the process during which learners 
interpret, translate or summarize the infonnation provided during the course and, 
consequentially, demonstrate the understanding of the learning event (Brodie, 2007). 
While the initially analyzed learning indicators have suggest positive effects of creating a 
portfolio as the learning indicators each contribute to improved comprehension, Table 8 
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directly seeks to identify the difference between online graphic design students 
participating in portfolio creation and the control (students not participating). 

Comprehension grades were detennined based on control and participant scores 
for rubrics designed to measure comprehension of design concepts and comprehension of 
design skills. The instructor computed participants' and control's final grades by 
considering the rubric designed and selected for this study. The highest comprehension 
grade, A, denoted online graphic design students that scored 5 points individual 
comprehension tests. While 85% of the participants (those creating a portfolio) scored A 
grades at the end of the seven week period, 50% of the control (non-participants) scored 
the A grade. This means that the number of participants in the portfolio creation that 
scored an A were 50% above non-participating counterparts. The same percentage (15%) 
of participants and control scored a B. 5% of the control scored an F in the 
comprehension test for online the graphic design course while no participant creating a 
portfolio scored below a B grade in the same comprehension tests. The results of the 
comprehension tests performed after seven weeks support Dewey’s educational 
philosophy that portfolio assessments promote real-world learning and help adult learners 
in distance education to master the new knowledge through reflective learning (Conrad, 
2008). Reflective learning in this case may be viewed as creating a portfolio since a 
portfolio refers to “a specific collection of material and documents, with the purpose of 
documenting a specific range of performance over a period of time . . . [and] . . . serves as 
a component of self-evaluation and provides participants and outsiders with an overview 
of personal learning success” (Zawacki-Richter et ah, 2011, p.45). 
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The students' final grade scored after five weeks is a reflection of the 
effectiveness of the different learning methods applied for the participant and control 
groups respectively. The final grade indicates the effects of the various learning 
indicators considered significant and applicable in this research study. The 8 reflects the 
final grades scored by the sample students offering the online graphic design course and 
can be used to compare the overall effect of participating in portfolio creation program 
and not participating in the program. With 50% of the participants scoring an A grade, 
students creating a portfolio presented a general A grade twice as much as the number of 
As presented by the control (non-participants). Also, while no participating failed the 
online graphic design course at week 5, 40% of the control failed the same course at 
Week 5. 45% participants scored B whereas only 25% control scored a B. The difference 
in the final grade results between the participants and the control was overwhelming and 
clearly suggests that creating a portfolio positively influences comprehension, self- 
efficacy and student-interaction form improved performance in an online graphic design 
learning environment. 

Summary of Findings 

Although this research could significantly contribute to the knowledge of 
pedagogy particularly in the ever-developing area of online based distance learning, it is 
important to note that the findings in this research may not be directly generalizable as it 
gives room for further extensive research studies. For instance, extensive research may be 
conducted using the same methodology however, the research may focus on other factors 
influencing and indicating students' performance apart from perception of self-efficacy 
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and comprehension. This will expand the theoretical background of the research study 
and also increase the potentiality for additional findings. 

The lack of consideration of the two samples based on social factors also gives 
room for additional research. While this study simply concentrated on the effects of 
creating a portfolio for online graphic design courses, further research may consider 
social factors such as the gender effects associated with creating a portfolio on online 
distance education. Although the general effects identified in this research presently 
indicate that creating a portfolio is a viable option to improve the perceived self-efficacy, 
comprehension and final grades of online graphic design students, expanding the sample 
would further increase knowledge of pedagogy as related to creating a portfolio. 

There are also some limitations associated with this research study. One 
significant limitation is the possibility of participant bias. There is a possibility that 
students were not straightforward when completing the surveys. This limitation was 
minimized by explaining the importance of the research study to all participants and 
encouraging students to be forthcoming when responding and assuring them that the 
instructor/researcher was not informed of their participation status. 

The selection of the sample for the survey may have also affected the results. This 
is because not all the students engaged in the online distance learning graphic design 
courses were included in the research study; with a sample of 20 and a control of 20, it is 
possible that the actual potential scores meant to be represented by the participants was 
not reflected and adversely affected the results of the study. 

This section of the report presented an analysis of the data collected in the 
research study. The data derived through the primary research was interpreted in-line 
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with the different theoretical ideologies and findings in the background section and 
literature review of the study. The findings basically suggested that creating a portfolio 
positively influenced online graphic students' academic perfonnance in terms of 
perceived self-efficacy and comprehension. 
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CHAPTER 5. RESULTS, CONCLUSIONS, AND RECOMMENDATIONS 


Introduction 

The purpose of this study was to analyze and assess how students in an online 
graphic design course would respond to creating a portfolio of their weekly designs. Also 
studied were students’ perceived self-efficacy, comprehension levels of the required 
software and a comparative case study was presented to show the differences of final 
grades between the control group and the participants of this study. G.T. Brown (2008) 
affirmed that portfolios first began in higher education 25 years ago and since then 
portfolios have been used within almost all disciplines (Klenowski et ah, 2006; Kuper et 
al., 2007; Snadden & Thomas, 1998; Tochel et ah, 2009). The importance of portfolios 
has been demonstrated through being used within the health and medical fields (Grant et 
al., 2007), teacher education (Chetcuti, 2008), as an instrument of literacy assessment 
(Walsh, 2009) and also used as an assessment tool for learning a second language 
(Cummins & Davesne, 2009). 

Previous research demonstrated the need and the importance of using portfolios 
for the online (distance education) environment for science and math courses 
(Drummond, 2004). Where research was lacking until this study; was determining the 
importance of creating and using portfolios within design courses within an online or 
distance educational setting. Portfolios as an instrument of assessing learning reflect “a 
shift from a stress on individual responsibility for learning to a more collaborative view, 
allowing learners to identify issues in their organization and society which affect their 
learning and well-being” (Klenowski et al., 2006, p.269). It is through the prism of the 
constructivist philosophy that the link between portfolio assessment and students’ self- 
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efficacy can be better understood. Through this study, an insight into the effects of 
portfolios on self-efficacy in learning was provided through the data analysis. 

Understanding the relationship between self-efficacy and portfolio assessments is 
important, since higher learner self-efficacy beliefs are fundamental to the professional 
maturation of students (Jones, 2009). Learner self-efficacy further predetennines higher 
motivation and better learning outcome expectations (Jones, 2009). Self-efficacy in 
learners is closely related to personal responsibility, effective goal setting and 
transformations: Jones (2009) suggests that self-efficient learners are transfonnative 
learners, who assume personal control over their learning progress. Through the prism of 
Bandura and Mezirow’s theories, self-efficacy drives the rapid transformation of the 
learning process, turning learners into both producers and products of their social 
environment (Jones, 2009). 

Summary of the Findings 

The purpose of this study is based on real life observations and supported by 
empirical evidence collected through both - the data collection phase and the data 
analysis of an online graphic design Digital Imaging course. Through the completion of 
this comparative case study, this study was able to provide positive responses to each of 
the research questions and, is surmised that creating portfolios is successful and has been 
found to successfully impact comprehension skills and self-efficacy of online students. 
According to the first research where students were sub divided into manageable groups 
and tested on their ability to make online graphical designs - it was proven that portfolios 
helped in not only developing but also improving self-efficiency and comprehension. The 
research was conducted in a classical environment where all students were exposed to the 
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same academic environment as a constant. In addition, data collected proved that 
portfolio assessment facilities utilized concept analysis thus increasing the level of self- 
efficiency demonstrated by the design students. 

In analyzing these results, it was found that a portfolio assessment enabled 
participants to develop a progressive attitude towards self-enhancement because the 
experiments associated portfolios with the creation of mental images among its users. 
These images are practical in fostering cognitive skills in the users because they create a 
lasting impression on the brains of the students thus facilitating communication as well as 
phenomenological understanding of scientific events. Furthermore, the results collected 
from the experiments were in line with Mezirow’s Transformative Learning theory which 
supports that human beings often use frames of mind to telegraphically transfer mental 
images into practical mechanical output where output is a measurable productivity index 
(Edirippulige & Mrasinghe, 2011). Comprehensively, portfolios magnify output to 
suitable amplitudes that can indomitably be reflected by the level of a person’s economic 
achievements, self-efficacy and personal degree of comprehension. 

In comparing the two groups; the participants and the control study groups on the 
basis of participant and non-participant courses, the findings are consistent with those 
made by Fisher (2003) in the book “Online collaborative learning: Relating theory to 
practice. Journal of Educational Technology Systems ” where he associated portfolios 
with a continuous assessment process which is essential in tracking long periods of 
teaching against that made during a single observation. He further described portfolios as 
a bridge that connects both, the processes and the products. On the other hand, it was 
discovered that portfolios could be helpful in institutionalizing norms related to reflection 
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of past events, as well as promote collaborative mechanisms between students. After the 
research, it was discovered that the portfolio process has many folds of benefits not only 
to the students but also to their respective institutions of higher learning since they 
provide an organizational structure through which other administrative functions can be 
undertaken. 

Lastly, the successes of portfolios can be attributed to a constructivist type of 
learning where education instructors shift teaching responsibilities to participatory 
learning which is more practical. These studies which were conducted using standard 
parameters while other factors were kept constant gives the most suitable findings which 
can be a pointer towards making a comprehensive summary on the findings extracted 
from the statistical data collected. Majorly, the study is comprised of data from online 
graphic design classes therefore there is a high probability of adherence to principles of 
objectivity. 

In summarizing our findings, the report will not be complete without mentioning 
the main impacts of portfolios on students as it was identified in the study survey. Now 
that this question carries the underlying points that support the research questions within 
this study, it is worth noting that a student’s success in the online design courses is 
detennined by time factor. Time is a constant resource which is equally available to 
everyone but some people spend their time more constructively than others. For instance, 
it was established that students who spend more time reading and making preparations 
about works assigned to them posted better results than those who invested little time in 
reading. 
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It has been proven that harder courses require more resource input than easy ones 
especially when the parties involved did not anticipate for such challenges. At first, the 
students are more dependent on the instructor for advice and support services but as time 
passes, the instructor-centered system of learning is slowly encroached by perceived self- 
efficacy from participants. From the analysis provided, it will be summative enough to 
note that portfolios have a profound impact on self-efficacy that is often boosted by the 
self-esteem levels of the participants. Secondly, the students using this system have been 
identified to benefit in tenns of comprehensive or cognitive capacity of participants. 

At the end of the day, a course instructor will only have to provide positive, 
formative and infonnative feedback to the students thus portfolio assessment has the 
characteristic of demonstrating the perception of self efficacy and student 
comprehension. These aspects can be gauged using a control experiment where 
environmental factors can be adjusted to show a slight variation in data collected for both 
comprehension and final grades. Data collected under the two conditions will portray a 
variation, which will be helpful in detennining forces that influence a more positive, 
academic performance. The data is then analyzed in comparison to a control experiment 
which has the ultimate outcome of determining indicators that interplay to influence 
efficiency and the overall performance of the students. 

Electronic Learning 

In conclusion, Information Technology is fast paced towards changing the face of 
the earth. Humanity is being transfonned day after day in the aspect that people are 
becoming polarized towards using technology in such sectors as economics and 
academics. Electronic commerce is one application of the use of the internet and 
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technological gadgetry to transact through a given commercial platform. Likewise, 
institutions of higher learning are advocating for the introduction of technology and 
relevant facilities that will fuel full adoption of electronic learning. Electronic learning 
can be acronym to e-learning where students use web-based pages to access information 
about their universities, colleges and schools from the comfort of their homes. 

It is worth noting that information collected by this research is not a 
generalization therefore the researcher is not subjective towards promoting or selling of 
her ideology on portfolios. The truth is that the research has been done extensively to 
cover any shortcomings that might emerge in the near future. On the other hand, the 
subject matter under study is so wide that it is likely to lead to controversies since some 
underlying principles are related. In that case, this dissertation is fashioned in such a 
manner that is suitable to comprehend. Specifically, the study will be of much help to 
pedagogies trying to establish how portfolios can impact student’s self-efficacy and 
comprehension in an online graphic design course. The two pertinent factors in focus are 
self-efficacy and comprehension and how they impact student’s perfonnance. It is also 
certain that this study is completely theoretical and thus it is purely premised on 
qualitative statistical methods. This is because qualitative research as a data collection 
tool provides more empirical facts about our subject of study. 

Limitations of the Study 

Limitations define shortcomings identified during the research process. A general 
limitation towards conducting this dissertation is that the subject area in question is very 
wide in that not a single research can cover all the topics. According to articles cited in 
the literature review, new models are being developed to counter the growing demand for 
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more quality learning institutions and educational facilities. Because of such pressures, 
there has been a remarkable shift towards perfonnance oriented measures of education 
viability. Above all the changes and transformation, many national governments have 
requested education officers to ensure that portfolio assessments embrace high degree of 
integrity, accountability and transparency. Despite such appeals by the government, 
nothing much has been done since most institutions of higher learning are lingering 
behind. This has led to provisions of sub-standard services that cannot match up to 
international standards, which are measured by ISO certification of organizations. 

Another limitation in conducting portfolio assessment research is that different 
educational instructors involved in the research had different scaling of marks since 
portfolio assessment is dependent on assessment methods used. This is a challenge and a 
bottleneck since detennining a neutral point in marks; allocation could be tricky. The 
limitation is fueled further by socio-political factors since the learners might not get an 
opportunity to meet and interact with the instructors; therefore it develops an intricate 
chain of complex conditions of online knowledge delivery. The end result is that 
transparency is obscured consequently transforming a well meant project into disaster. 
For the shortcomings to be streamlined, governments should work hand in hand with 
education officers in order to create insight into the utilization and usability of given 
assessment procedures. 

An additional limitation noted while undertaking this study is that research on 
portfolio assessments especially for an online graphic design course is that they are 
constantly increasing in scope of coverage. Because of this reason there exists a literature 
gap that would help link the use of portfolios in e-learning to self-efficacy and 
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comprehension. To some extent the literature gap has worked to the advantage of this 
research since it is focused on assessing the possibility of success in a competence based 
learning environment. Through this study, it has become easy for educationists to 
establish the growth matrix in terms of popularity of reflective portfolio assessment in 
institutions of higher learning. In conducting this research therefore, due attention was 
taken to employ the most suitable methodology in establishing effects accrued from 
comprehension and evaluation of the perceived self-efficacy of students. 

Currently there is a wealth of information on portfolio assessments but most areas 
have been covered extensively which means that there are lots of generalizations. This is 
a limiting factor in conducting a study based on self efficiency and competence since 
different researchers envelop this topic from a diverse perspective covering such topics as 
the role of portfolio in assessment, criteria for using portfolios, characteristics and 
qualities of good portfolios which, in the absence of the qualitative angle, fails to 
integrate portfolios with the sequence of learning experiences leading students towards 
the desired goal. However this limitation has been catered for in this study by employing 
the use of qualitative methods of research, which provides conclusive data about self- 
efficacy and comprehension. 

Other general limitations experienced during this research is that the samples used 
represented a graphic design course and which may not necessarily represent all 
undergraduate courses offered within institutions of higher learning in the United States. 
Alternatively, student opinions voiced by 20 participants may not be sufficient to 
represent those of a different course given the dynamics related to associates degree 
program for the online graphic design degree program. Given the complexity of graphic 
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deigning, some participants opted to exit the program and concentrate on other activities. 
Several surveys conducted could have marginal errors especially when the surveyors 
were subjective in recording their findings. 

Emerging Themes Through Data Analysis 
Data analysis is an extensive, statistical practice that involves the comparison of 
recorded facts and figures in order to infer on a given subject matter. Given this research 
that is devoted to establishing how portfolios can impact student’s self-efficacy and 
comprehension in an online graphic design course, then there is need for concept 
analysis. Concept analysis will embrace a theme allied to giving of a summative idea on 
observations or experiences as recorded by the sample population used for the study. This 
is only possible with practice which aid in developing superb cognitive skills that are 
guiding to an analyst. 

Cognitive skills can enhance creation of mental images, which have been 
accredited to facilitating comprehensible communication and expression of ideologies. 
This aspect has been described by Mezirow’s Transfonnative Learning Theory as a 
process entailing the use of a “frame of mind” where mental images are used to the 
advantage of the researcher to enhance his or her breakthrough in the research (Butler & 
Winne, 2010). This is in itself commendable since it is an indicator of how practical 
personal levels of self-efficacy and comprehension can be if applied in an online graphic 
design study. Most times, data analysis encompasses comparisons as a theme. In our case 
comparison can be made for both participants and non-participants. This proposal has 
been supported by Shulman (2003; as cited in Lombardi, 2008), who purports that 
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portfolio assessment who has implicitly stated his liking for constructive learning in 
institutions of higher learning. 

In synthesizing this data, an analyst ought to operate within the range of an 
average class size of 20 students undertaking their studies using an online platform. The 
course in question is a graphic design course although the research could be multi-faced 
in terms of coverage so as to cover other course units dealing in graphic design. The 
participants undertaking this study are associate degree program students where half of 
the sample population represents a control group. Analysis of the statistical figures 
collected from the study indicates a direct proportionate ratio between the amount of 
resource input and success of the project. An average student without experience on 
software programming is likely to spend a total of up to ten hours per week on design 
related studies. In this sense, it is worth observing that a challenging course will require 
more time input than a simple course. Similarly, introduction of subjects may require 
more time because introductions are meant to cover most general aspects of a topic thus 
making it a time consuming venture. 

Contrary to the observations made among the participant sample population, the 
control group exhibited a conservative use of the available time. There was also constant 
interaction between participants and the instructors. In the case of portfolios, the same 
scenario is implicit. According to the recorded observations made per the work progress 
sheet, each passing week is important in shaping the self-esteem of the students. The 
participants were noted to have improved their perception on self-efficacy and 
comprehension and cognitive abilities. 


129 



Discovery in Research 


In given research practices, there are certain implications that come with the 
reports made. There are benefit groups that find this infonnation very helpful since the 
researcher’s goal was to fill a certain gap in research, within a given discipline. The 
information contained in a research study is not for the person who wrote it but for the 
general public with the premise of providing research that will enhance the world of 
education. Thus there are principles such as that of empirical data and objectivity, which 
ensures that a study is accurate and as outstanding, as possible. There are ethical 
implications that come with conducting a research as well. For instance ethical 
considerations provide for confidentiality of the participants in both the select and control 
groups. Ethical implications of this research were challenged by the practicality of 
qualitative research. The qualitative aspect of the ethics code of conducting research was 
implemented using a written guarantee where participants were assured of privacy and no 
personal information would be provided to the public. 

In terms of content implications, the research was geared towards establishing 
how portfolios can impact student’s self-efficacy and comprehension in an online graphic 
design course. The collected results had the implication that portfolios are very relevant 
in promoting modem learning experiences in institutions of higher learning. The goal of 
higher education is actually to identify a student’s strengths and weaknesses, review and 
validate the effectiveness of various teaching strategies and infonn administrative and 
instructional decision making, portfolio assessments have all chances to become a viable 
mechanism of assessing learners’ progress in higher education. From the research it is 
implicit to say that portfolio assessment is a historical strategy that was implemented in 
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the United States. It had the preliminary objective of encouraging and empowering 
students to evaluate their knowledge and learning progress through cognitive abilities and 
critical thinking. The traditional portfolios have since then transformed into new 
technology based forms of assessing student’s knowledge. 

The main challenge associated with portfolio assessment is that many have 
misinterpreted its importance. Being a type of assessment, portfolio can be conducted 
through the internet therefore it is applicable to a wide array of disciplines of study such 
as medical, health, teacher assessment and more. Furthermore it is applicable in literacy 
development and graphic design courses although very few researchers have gone the 
extra mile of investing in undertaking studies related to online graphic design 
environments. This research therefore has the ultimate implication that the education 
system is moving away from process oriented to outcome oriented assessment 
philosophies. Technological advancement also changes the nature of knowledge delivery 
and need to re-evaluate applicability of portfolios in distance education. 

This research has the implication of filling in the gap that has been existent in 
empirical literature since its inauguration in the U.S since it justifies the need for 
analyzing the usefulness as well as the usability of portfolio assessments in online 
learning environments (Ormrod & Jeanne, 2009). Given that comprehensiveness and self- 
efficacy have been extensively explored in previous researches, the current state of 
available information materials suggests that constructivist learning can be integrated as a 
proffered network for professional educators who may use it to design learning solutions 
for their learners. It therefore confers that Bandura’s social cognitive theory becomes 
evident from the findings posted in this report. 
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Conclusion 


In conclusion, this research was a success in determining that portfolios can 
impact student’s self-efficacy and comprehension in an online graphic design course. The 
research is in line with constructivism theories which are established on the groundwork 
of reinterpreting the connectivity bond between self efficacy and comprehension. These 
findings therefore have the potential of improving professional understanding of 
educationists toward portfolio assessments. If all is said and done, then portfolios will act 
as a leeway into improving the quality of education through bettennent of distance 
learning, which is currently being done electronically (Joachim & Nadine 2009). 

Comprehension in students has been noted in Blooms’ learning taxonomy. The 
same theory has been proven in this study since students were discovered to have the 
ability to recognize, analyze and apply the new knowledge that can be applied in real life. 
The applicability of these findings are the very core of any leaner-centered nature of 
knowledge development and learning process as is the same case with portfolio 
assessment. In case this technique is embraced and modified to march future 
requirements in the education sector, then students will portray a more systematic and 
organized way of gaining knowledge and learning of new skills. This study will impact 
higher education and provide not only means to future research, but to explore other 
online classrooms to see if portfolios can indeed be used as a proper assessment tool. This 
study has also proven that while building on student’s self-efficacy that it does impact 
their comprehension levels within the online classroom. Finally, this study also helps 
bridge the gap between students and their instructors while fulfilling the need for research 
in this area of study. 
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APPENDIX A. SELF-EFFICACY SURVEY - CONTROL GROUP 


Course number: Design 160 
Course title: Digital Imaging 

On average, how many hours do you spend online (on Blackboard) for your course each 
week? 

Less than 5 hours 
6-10 hours 
11-15 hours 
16-20 hours 
Above 20 hours 

(Please mark the appropriate number on the scale below each statement.) 

Learner- instructor interactions: 

1 . I had numerous interactions with the instructor during this class, which aided me 
in the process improving my designs. 

2. I asked the instructor my questions through different electronic means such as 
email, discussion board, instant messaging tools, or phone. 

3. The instructor replied to my questions in a timely fashion. 

4. I replied to messages from my instructor. 

5. I received both positive feedback and informative feedback on how I could 
improve my designs. 

(Strongly disagree 1 2 3 4 5 N/A Strongly agree) 

Perceived Satisfaction (self-efficacy) within a design course 

6. I have used this software prior to this course, so there were no issues in 
completing this class. 

7. When I began this course, I felt confident that I would enjoy creating my design 
projects. 

8. During the process of creating my designs, I had to contact the instructor with 
questions that arose. 

9. During the process of creating my designs, I felt confident enough to problem 
solve on my own. 

10. 1 would recommend this process for all my design courses. 

(Strongly disagree 1 2 3 4 5 N/A Strongly agree) 

Open-ended questions to show progression of self-efficacy 
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1 1 . How would you compare your level of confidence from week one through week 
five and why? 

12. Did you notice any benefit of your instructor feedback for this design course? 

Why or why not? 

13. Now that the course has been completed, do you feel more confident in your skills 
and knowledge to move forward into your future design courses? Why or why 
not? 

14. What tips - if any - would you provide students entering into this design course 
and why? 
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APPENDIX B. SELF-EFFICACY SURVEY - PARTICIPANTS 


Course number: Design 160 
Course title: Digital Imaging 

On average, how many hours do you spend online (on Blackboard) for your course each 
week? 

Less than 5 hours 
6-10 hours 
11-15 hours 
16-20 hours 
above 20 hours 

(Please mark the appropriate number on the scale below each statement.) 

Learner- instructor interactions: 

1 . I had numerous interactions with the instructor during this class, which aided me 
in the process of creating a portfolio. 

2. I asked the instructor my questions through different electronic means such as 
email, discussion board, instant messaging tools, or phone. 

3. The instructor replied to my questions in a timely fashion. 

4. I replied to messages from my instructor. 

5. I received both positive feedback and informative feedback on how I could 
improve my portfolio. 

(Strongly disagree 1 2 3 4 5 N/A Strongly agree) 

Perceived Satisfaction (self-efficacy) with outcome of creating a portfolio 

6. I have created portfolios prior to this course, so there were no issues in 
completing this process. 

7. When I began this course, I felt confident that I would enjoy the process of 
creating a portfolio. 

8. During the process of creating my portfolio, I had to contact the instructor with 
questions that arose. 

9. During the process of creating my portfolio, I felt confident enough to problem 
solve on my own. 

10. 1 would recommend this process for all my design courses. 

(Strongly disagree 1 2 3 4 5 N/A Strongly agree) 

Open-ended questions to show progression of self-efficacy 

1 1 . How did creating the portfolio, increase your level of competence within the 
course of study? 

12. Did you notice any benefit of creating a portfolio for this design course? Why or 
why not? 
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13. What do you think of using a portfolio as a learning tool? Please be explicit. 

14. What do you think of using a portfolio as an assessment tool for this course? 

15. Now that the course and the portfolio have been completed, do you feel more 
confident in repeating this process within future design courses? Why or why not? 

16. How has your confidence level changed as a result of creating a portfolio and 
being able to see the progression of your work? Please be explicit. 
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APPENDIX C. RUBRIC - COMPREHENSION OF CONCEPTS IN DESIGN 


5 - Concepts of discussion have been successfully applied and developed with 
appropriate use of proper terminologies. Discussion posted before deadline. 

4 - Concepts have been developed successfully, but clarity is missing as to the proper 
tenninologies. Discussion posted on time. 

3 - Concepts have somewhat been developed with some use of proper tenninologies. 
Discussion posted during the week, but deadline not met. 

2 - Concepts not clear with minimal levels of tenninologies. Discussion is posted after 
the current week and past the deadline. 

1 - Concepts not clear and improper use of terminologies. Discussion submitted after 
allotted allowance for an assignment to be posted. 
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APPENDIX D. RUBRIC - COMPREHENSION OF SKILLS IN DESIGN 


5 - Proper techniques have been demonstrated by implementing required techniques into 
design work. Craftsmanship of the assignment will demonstrate technical abilities, 
comprehension of the subject matter applied to design work, plus demonstrate the ability 
to introduce new techniques into their designs. Subject matter will be presented in a 
professional manner. 

4 - Most techniques have been demonstrated by implementing some - but not all required 
techniques into design work. Craftsmanship of the assignment will demonstrate technical 
abilities, comprehension of the subject matter applied to design work, plus demonstrate 
the ability to introduce new techniques into their designs. Subject matter will be 
presented in a professional manner. 

3 - Limited techniques have been demonstrated by implementing some - but not all 
required techniques into design work. Craftsmanship of the assignment will demonstrate 
technical abilities; comprehension of the subject matter applied to design work, but did 
not apply any new techniques. Subject matter was presented in a less than professional 
manner. 

2 - A few techniques have been demonstrated by implementing some - but not all 
required techniques into design work. Craftsmanship of the assignment does not 
demonstrate technical abilities, and demonstrates a lack of comprehension of the subject 
matter applied to the design work. Subject matter was presented in a less than 
professional manner. 

1 - No techniques have been applied as required into the design work. Craftsmanship of 
the assignment does not demonstrate technical abilities, and demonstrates a lack of 
comprehension of the subject matter applied to the design work. Subject matter was 
presented in a less than professional manner. 
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